a:4:{s:5:"child";a:1:{s:0:"";a:1:{s:3:"rss";a:1:{i:0;a:6:{s:4:"data";s:3:" ";s:7:"attribs";a:1:{s:0:"";a:1:{s:7:"version";s:3:"2.0";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:1:{s:0:"";a:1:{s:7:"channel";a:1:{i:0;a:6:{s:4:"data";s:217:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:1:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:12:"Planet MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:27:"http://www.planetmysql.org/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Wed, 30 Jun 2010 09:15:01 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"language";a:1:{i:0;a:5:{s:4:"data";s:2:"en";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:42:"Planet MySQL - http://www.planetmysql.org/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"item";a:50:{i:0;a:6:{s:4:"data";s:88:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:54:"Making “Replace Into” Fast, by Avoiding Disk Seeks";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:26:"http://tokutek.com/?p=1548";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:75:"http://tokutek.com/2010/06/making-replace-into-fast-by-avoiding-disk-seeks/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:3211:" In this post two weeks ago, I explained why the semantics of normal ad-hoc insertions with a primary key are expensive because they require disk seeks on large data sets. Towards the end of the post, I claimed that it would be better to use “replace into” or “insert ignore” over normal inserts, because the semantics of these statements do NOT require disk seeks. In this post, I explain how the command “replace into” can be fast with fractal trees. The semantics of “replace into” are as follows: if the primary (or unique) key does not exist, insert the new row if the primary (or unique) key does exist, overwrite the existing row with the new row The slow, expensive way B-trees use to implement these semantics are: look up the primary (or unique key), to verify its existence if it does not exist, insert the new row, otherwise overwrite the existing row The first step incurs a disk seek. That slows down performance considerably. Instead, with TokuDB’s fractal tree data structure, we can follow a similar strategy to what we use for deletes. Recall that for deletes, we do not look up the key we wish to delete, and physically remove its data from the tree. Instead, we use tombstone messaging, or tombstone deletions, to defer the physical removal of the data to a better time. The same idea applies here. Instead of searching for the primary key, as we are forced to do with B-trees, we insert an insertion message into the fractal tree, and defer the existence check for later. Let us look at an example. Take the following table: create table foo (a int, b int, primary key (a)); Suppose the fractal tree for this table looks as follows: - - - - - - - .... (i (1,1)) (i (2,2)) (i (3,3)) (i (4,4)) ... (i (1000,1000)) ... (i (2^32, 2^32)) The ‘i’ stands for insertion message. Now suppose we do: replace into foo values (1000, 1001). With fractal trees, we simply insert (i (1000,1001)) into the top node. The tree then looks as such: (i (1000,1001)) - - - - - - .... (i (1,1)) (i (2,2)) (i (3,3)) (i (4,4)) ... (i (2^32, 2^32)) Similar to deletes, with this scheme, “replace into” can be two orders of magnitude faster than insertions into a B-tree. On queries, a message in a higher node overrides messages in lower nodes. So upon querying the key ’1000′, a cursor notices that (1000,1001) is located higher than (1000,1000), and therefore returns (1000,1001) to the user. On merges, the message in the higher node overwrites the message in the lower node. So, by using messages, fractal trees can achieve the same performance boost for “replace into” as it does for insertions. In fact, using “replace into” in the manner above is how a customer has achieved an 80x speedup under actual field conditions. The details can be found here. Next week, I explain how “insert ignore” can similarly be fast. Also, while this shows how “replace into” can be fast (and IS fast for a lot of scenarios we see with TokuDB), there are some caveats (with indexes, triggers, and replication). I will get into those in a couple of weeks.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Wed, 30 Jun 2010 04:10:02 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:10:{i:0;a:5:{s:4:"data";s:8:"TokuView";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:6:"B-tree";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:9:"disk seek";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:13:"Fractal Trees";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:6:"insert";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:7:"replace";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:12:"replace into";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:6:"TokuDB";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:9;a:5:{s:4:"data";s:6:"update";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:4008:"

In this post two weeks ago, I explained why the semantics of normal ad-hoc insertions with a primary key are expensive because they require disk seeks on large data sets. Towards the end of the post, I claimed that it would be better to use “replace into” or “insert ignore” over normal inserts, because the semantics of these statements do NOT require disk seeks. In this post, I explain how the command “replace into” can be fast with fractal trees.

The semantics of “replace into” are as follows:

The slow, expensive way B-trees use to implement these semantics are:

The first step incurs a disk seek. That slows down performance considerably.

Instead, with TokuDB’s fractal tree data structure, we can follow a similar strategy to what we use for deletes. Recall that for deletes, we do not look up the key we wish to delete, and physically remove its data from the tree. Instead, we use tombstone messaging, or tombstone deletions, to defer the physical removal of the data to a better time. The same idea applies here. Instead of searching for the primary key, as we are forced to do with B-trees, we insert an insertion message into the fractal tree, and defer the existence check for later.

Let us look at an example. Take the following table:

create table foo (a int, b int, primary key (a));

Suppose the fractal tree for this table looks as follows:

- 

- -

- - - -

....

(i (1,1)) (i (2,2)) (i (3,3)) (i (4,4)) ... (i (1000,1000)) ... (i (2^32, 2^32))

The ‘i’ stands for insertion message. Now suppose we do:

replace into foo values (1000, 1001).

With fractal trees, we simply insert (i (1000,1001)) into the top node. The tree then looks as such:

(i (1000,1001)) 

- -

- - - -

....

(i (1,1)) (i (2,2)) (i (3,3)) (i (4,4)) ... (i (2^32, 2^32))

Similar to deletes, with this scheme, “replace into” can be two orders of magnitude faster than insertions into a B-tree.

On queries, a message in a higher node overrides messages in lower nodes. So upon querying the key ’1000′, a cursor notices that (1000,1001) is located higher than (1000,1000), and therefore returns (1000,1001) to the user. On merges, the message in the higher node overwrites the message in the lower node.

So, by using messages, fractal trees can achieve the same performance boost for “replace into” as it does for insertions. In fact, using “replace into” in the manner above is how a customer has achieved an 80x speedup under actual field conditions. The details can be found here.

Next week, I explain how “insert ignore” can similarly be fast. Also, while this shows how “replace into” can be fast (and IS fast for a lot of scenarios we see with TokuDB), there are some caveats (with indexes, triggers, and replication). I will get into those in a couple of weeks.


PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Tokuview Blog";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:1;a:6:{s:4:"data";s:43:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:19:"SQL past and future";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:70:"tag:blogger.com,1999:blog-5672165237896126100.post-4814397995264524607";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:63:"http://julianhyde.blogspot.com/2010/06/sql-past-and-future.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:3184:"Ken North, writing in Dr. Dobb's Journal, gives a nice overview of the long and storied history of SQL. The piece helps one understand the wave of mergers among the big database vendors, and make sense of current trends in database and database-like software. And I'd like to offer my opinion about where SQL and database management systems are headed.North looks into the claims that 'the database is dead' and finds that — yet again — reports of its death were greatly exaggerated:Forrester Research recently estimated the total database market (licenses, support, consulting) would grow from $27 billion in 2009 to $32 billion by 2012. SQL technology is entrenched in many organizations and across millions of web sites. Perhaps that explains why, during the past decade, IBM, Oracle, Sun and SAP made billion-dollar investments in a ‘dead’ technology.However, I do believe that the relational database is currently in crisis. Relational databases have been the mainstay of data management for over twenty years, but Oracle and its cohorts have no answer for "Big Data", the massive onslaught of information from the web and sensors.The NoSQL movement is solving these problems by challenging some of the assumptions held by RDBMS vendors. At SQLstream, we regard ourselves as part of the NoSQL movement even though we are huge fans of SQL, because we are challenging the biggest assumption of them all: that you have to put data on disk before you can analyze it.It's a shame that North doesn't mention streaming SQL, because it fits perfectly into the grand arc of the SQL language: adopt new problems, express them declaratively, and solve them first with special-purpose database engines and finally by adapting the architecture of the big, general-purpose database engines. This last step sometimes takes many years to happen, but it happened for transaction processing, object database, and data warehousing, and I have no doubt that it will happen for streaming relational data.One of the reasons that SQL has remained relevant is SQL standards process; products built on one database can be run on another database and, perhaps more important, skill sets acquired on one engine can be applied to another. When the dust settles, and the big databases have learned hard architectural lessons, I think a lot of these new problems will be solved in SQL. Unlike Mike Stonebraker, I do think that organizations will want to put all these different forms of data into one database management system. That database will of course be a facade spread over many servers, disks, data organizations and query processing engines, but will offer centralized management and allow the different forms of data to be combined. They will get their wish because the SQL language is so powerful at hiding differences in underlying data organization.When the dust has settled, the SQL language will have changed and adapted yet again, and maybe there will be some new names at the top of the roster of database vendors, but we will once again be solving most of our data management problems using declarative queries beginning with the word "SELECT". SQL is dead; long live SQL!";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Wed, 30 Jun 2010 01:08:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:1:{i:0;a:5:{s:4:"data";s:50:"sql oracle db2 nosql "one size fits all"";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:3967:"Ken North, writing in Dr. Dobb's Journal, gives a nice overview of the long and storied history of SQL. The piece helps one understand the wave of mergers among the big database vendors, and make sense of current trends in database and database-like software. And I'd like to offer my opinion about where SQL and database management systems are headed.

North looks into the claims that 'the database is dead' and finds that — yet again — reports of its death were greatly exaggerated:
Forrester Research recently estimated the total database market (licenses, support, consulting) would grow from $27 billion in 2009 to $32 billion by 2012. SQL technology is entrenched in many organizations and across millions of web sites. Perhaps that explains why, during the past decade, IBM, Oracle, Sun and SAP made billion-dollar investments in a ‘dead’ technology.
However, I do believe that the relational database is currently in crisis. Relational databases have been the mainstay of data management for over twenty years, but Oracle and its cohorts have no answer for "Big Data", the massive onslaught of information from the web and sensors.

The NoSQL movement is solving these problems by challenging some of the assumptions held by RDBMS vendors. At SQLstream, we regard ourselves as part of the NoSQL movement even though we are huge fans of SQL, because we are challenging the biggest assumption of them all: that you have to put data on disk before you can analyze it.

It's a shame that North doesn't mention streaming SQL, because it fits perfectly into the grand arc of the SQL language: adopt new problems, express them declaratively, and solve them first with special-purpose database engines and finally by adapting the architecture of the big, general-purpose database engines. This last step sometimes takes many years to happen, but it happened for transaction processing, object database, and data warehousing, and I have no doubt that it will happen for streaming relational data.

One of the reasons that SQL has remained relevant is SQL standards process; products built on one database can be run on another database and, perhaps more important, skill sets acquired on one engine can be applied to another. When the dust settles, and the big databases have learned hard architectural lessons, I think a lot of these new problems will be solved in SQL. 

Unlike Mike Stonebraker, I do think that organizations will want to put all these different forms of data into one database management system. That database will of course be a facade spread over many servers, disks, data organizations and query processing engines, but will offer centralized management and allow the different forms of data to be combined. They will get their wish because the SQL language is so powerful at hiding differences in underlying data organization.

When the dust has settled, the SQL language will have changed and adapted yet again, and maybe there will be some new names at the top of the roster of database vendors, but we will once again be solving most of our data management problems using declarative queries beginning with the word "SELECT". SQL is dead; long live SQL!

PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:11:"Julian Hyde";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:2;a:6:{s:4:"data";s:168:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:25:"451 CAOS Links 2010.06.29";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:47:"http://blogs.the451group.com/opensource/?p=1973";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:60:"http://feedproxy.google.com/~r/451opensource/~3/Uj8zngkl0S0/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:2445:"Elephants on parade: Hadoop goes mainstream. And more. Follow 451 CAOS Links live @caostheory on Twitter and Identi.ca “Tracking the open source news wires, so you don’t have to.” Elephants on parade # Cloudera launched v3 of its Distribution for Hadoop and released v1 of Cloudera Enterprise. # Karmasphere released new Professional and Analyst Editions of its Hadoop development and deployment studio. # Talend announced that its Integration Suite now offers native support for Hadoop. # Yahoo announced the beta release of Hadoop with Security and Oozie, Yahoo’s workflow engine for Hadoop. # Datameer announced a strategic partnership with Zementis for predictive analytics on Hadoop. # The Register reported that Twitter is set to open source its MySQL-to-Hadoop tool. # MicroStrategy announced support for Apache Hadoop as a data source for MicroStrategy 9. # Appistry announced Hadoop-based strategic alliances Concurrent, Datameer and Kitenga. # GOTO Metrics released Data Analytics Platform, a Hadoop-based business intelligence platform. Best of the rest # The Software Freedom Law Center responded to the Supreme Court’s decision on Bilski v. Kappos, while Mark Radcliffe provided his thoughts. # David Wiley discussed openness, radicalism, and tolerance (and the lack of it). # Jorg Janke discussed how Compiere overstepped the balance between proprietary and open product components. # Simon Phipps argued that open core is bad for software freedom. # Nick Halsey joined SugarCRM as chief marketing officer. # DotNetNuke more than doubled its subscription customers in 1H10 to nearly 800, expects 400% FY revenue growth. # Nuxeo announced its new Nuxeo Case Management Framework. # Mike Masnick discussed why the lack of billion dollar pure play open source software companies is a good thing. # The Apache Software Foundation announced Apache Tomcat Version 7.0. # Glyn Moody asked whether Oracle has been a disaster for Sun’s open source. # Infoworld discussed eight business strategies for profiting from open source software. # Computerworld reported that Red Hat CEO sees VMware as biggest competitor. # IBM published an essay on the role Linux plays in its smarter planet initiative. # Groklaw asked, What did Microsoft know about SCO’s plan to attack Linux, and when did it know it? # Mozilla won the American Business Award for the most innovative company of the year. ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Wed, 30 Jun 2010 00:14:42 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:26:{i:0;a:5:{s:4:"data";s:5:"Links";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:9:"451 group";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:13:"451caostheory";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:8:"451group";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:6:"apache";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:8:"appistry";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:6:"bilski";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:10:"caostheory";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:8:"cloudera";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:9;a:5:{s:4:"data";s:8:"compiere";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:10;a:5:{s:4:"data";s:10:"concurrent";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:11;a:5:{s:4:"data";s:8:"datameer";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:12;a:5:{s:4:"data";s:11:"david wiley";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:13;a:5:{s:4:"data";s:10:"DotNetNuke";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:14;a:5:{s:4:"data";s:12:"goto metrics";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:15;a:5:{s:4:"data";s:7:"groklaw";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:16;a:5:{s:4:"data";s:6:"Hadoop";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:17;a:5:{s:4:"data";s:10:"jorg janke";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:18;a:5:{s:4:"data";s:11:"karmasphere";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:19;a:5:{s:4:"data";s:7:"kitenga";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:20;a:5:{s:4:"data";s:5:"Linux";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:21;a:5:{s:4:"data";s:14:"Mark Radcliffe";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:22;a:5:{s:4:"data";s:11:"matt aslett";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:23;a:5:{s:4:"data";s:10:"mattaslett";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:24;a:5:{s:4:"data";s:14:"matthew aslett";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:25;a:5:{s:4:"data";s:3:"mat";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:3991:"

Elephants on parade: Hadoop goes mainstream. And more.

Follow 451 CAOS Links live @caostheory on Twitter and Identi.ca
“Tracking the open source news wires, so you don’t have to.”

Elephants on parade
# Cloudera launched v3 of its Distribution for Hadoop and released v1 of Cloudera Enterprise.

# Karmasphere released new Professional and Analyst Editions of its Hadoop development and deployment studio.

# Talend announced that its Integration Suite now offers native support for Hadoop.

# Yahoo announced the beta release of Hadoop with Security and Oozie, Yahoo’s workflow engine for Hadoop.

# Datameer announced a strategic partnership with Zementis for predictive analytics on Hadoop.

# The Register reported that Twitter is set to open source its MySQL-to-Hadoop tool.

# MicroStrategy announced support for Apache Hadoop as a data source for MicroStrategy 9.

# Appistry announced Hadoop-based strategic alliances Concurrent, Datameer and Kitenga.

# GOTO Metrics released Data Analytics Platform, a Hadoop-based business intelligence platform.

Best of the rest
# The Software Freedom Law Center responded to the Supreme Court’s decision on Bilski v. Kappos, while Mark Radcliffe provided his thoughts.

# David Wiley discussed openness, radicalism, and tolerance (and the lack of it).

# Jorg Janke discussed how Compiere overstepped the balance between proprietary and open product components.

# Simon Phipps argued that open core is bad for software freedom.

# Nick Halsey joined SugarCRM as chief marketing officer.

# DotNetNuke more than doubled its subscription customers in 1H10 to nearly 800, expects 400% FY revenue growth.

# Nuxeo announced its new Nuxeo Case Management Framework.

# Mike Masnick discussed why the lack of billion dollar pure play open source software companies is a good thing.

# The Apache Software Foundation announced Apache Tomcat Version 7.0.

# Glyn Moody asked whether Oracle has been a disaster for Sun’s open source.

# Infoworld discussed eight business strategies for profiting from open source software.

# Computerworld reported that Red Hat CEO sees VMware as biggest competitor.

# IBM published an essay on the role Linux plays in its smarter planet initiative.

# Groklaw asked, What did Microsoft know about SCO’s plan to attack Linux, and when did it know it?

# Mozilla won the American Business Award for the most innovative company of the year.


PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"The 451 Group";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:3;a:6:{s:4:"data";s:83:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:80:"Debugging InnoDB Locks using the new InnoDB Plugin’s Information Schema Tables";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:34:"http://www.chriscalender.com/?p=91";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:34:"http://www.chriscalender.com/?p=91";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:5534:"Tracking down InnoDB lock information using the new Information Schema tables provided with the InnoDB plugin has never been easier. Long story short, the other day I was trying to identify what transaction was holding the lock for a particular UPDATE. The UPDATE would not complete and kept timing out with “Lock wait timeout exceeded; try restarting transaction”. Of course I checked the output of SHOW ENGINE INNODB STATUS. From that output, I could tell it was the replication thread holding the lock, *since* it was the only other transaction running. But I could not verify this with output, it just had to be the case. Next I enabled the InnoDB lock monitor, and examined that output too. Here, it showed more detail on the locks being held, however, it only shows the first 10 locks held by the replication thread. In this case, the replication thread was holding more than 2000 record locks from a large, open transaction. Needless to say, the lock I was looking for was not amongst the 10 displayed. So I enabled InnoDB plugin (just added the following 2 lines to my config file and restarted mysqld): ignore_builtin_innodb plugin-load=innodb=ha_innodb_plugin.dll;innodb_trx=ha_innodb_plugin.dll; innodb_locks=ha_innodb_plugin.dll;innodb_lock_waits=ha_innodb_plugin.dll; innodb_cmp=ha_innodb_plugin.dll;innodb_cmp_reset=ha_innodb_plugin.dll; innodb_cmpmem=ha_innodb_plugin.dll;innodb_cmpmem_reset=ha_innodb_plugin.dll After the above, you’ll notice the following new tables in Information Schema: mysql> SHOW TABLES IN INFORMATION_SCHEMA LIKE 'INNODB_%'; +-----------------------------------------+ | Tables_in_information_schema (INNODB_%) | +-----------------------------------------+ | INNODB_CMP_RESET | | INNODB_TRX | | INNODB_CMPMEM_RESET | | INNODB_LOCK_WAITS | | INNODB_CMPMEM | | INNODB_CMP | | INNODB_LOCKS | +-----------------------------------------+ 7 rows in set (0.00 sec) For this particular problem, I wanted to see what was in INNODB_TRX, INNODB_LOCKS, and INNODB_LOCK_WAITS: mysql> SELECT * FROM INFORMATION_SCHEMA.INNODB_TRX; +--------+-----------+---------------------+-----------------------+---------------------+------------+---------------------+-----------+ | trx_id | trx_state | trx_started | trx_requested_lock_id | trx_wait_started | trx_weight | trx_mysql_thread_id | trx_query | +--------+-----------+---------------------+-----------------------+---------------------+------------+---------------------+-----------+ | 124616 | LOCK WAIT | 2010-06-23 13:51:52 | 124616:1301:1572:6 | 2010-06-23 13:51:52 | 2 | 496854 | NULL | | 123900 | RUNNING | 2010-06-23 09:30:59 | NULL | NULL | 506 | 3 | NULL | +--------+-----------+---------------------+-----------------------+---------------------+------------+---------------------+-----------+ 2 rows in set (0.00 sec) mysql> SELECT * FROM INFORMATION_SCHEMA.INNODB_LOCKS; +--------------------+-------------+-----------+-----------+-------------------------+------------+------------+-----------+----------+-----------+ | lock_id | lock_trx_id | lock_mode | lock_type | lock_table | lock_index | lock_space | lock_page | lock_rec | lock_data | +--------------------+-------------+-----------+-----------+-------------------------+------------+------------+-----------+----------+-----------+ | 124616:1301:1572:6 | 124616 | X | RECORD | `db1`.`t1` | `PRIMARY` | 1301 | 1572 | 6 | 4100 | | 123900:1301:1572:6 | 123900 | S | RECORD | `db1`.`t1` | `PRIMARY` | 1301 | 1572 | 6 | 4100 | +--------------------+-------------+-----------+-----------+-------------------------+------------+------------+-----------+----------+-----------+ 2 rows in set (0.01 sec) mysql> SELECT * FROM INFORMATION_SCHEMA.INNODB_LOCK_WAITS; +-------------------+--------------------+-----------------+--------------------+ | requesting_trx_id | requested_lock_id | blocking_trx_id | blocking_lock_id | +-------------------+--------------------+-----------------+--------------------+ | 124616 | 124616:1301:1572:6 | 123900 | 123900:1301:1572:6 | +-------------------+--------------------+-----------------+--------------------+ 1 row in set (0.00 sec) Finally, I had conclusive evidence that it was indeed the replication thread (trx id 123900) holding the lock which was blocking the UPDATE. I also found the exact row which was being held, and all of the details, such as it was a S (shared) record lock, and the the lock was on the primary key of table db1.t1. It even tells you the lock_space (1301), the lock_page (1572), the lock_rec (6) , and the actual data, lock_data (4100). The other key piece was the time (or trx_started) for the running transaction, 123900. With that, we knew exactly where to check the binlogs for the exact statement where this all began. In conclusion, I wish I had started here to begin with, so keep this in mind for the future. And this is just one example of how beneficial these new InnoDB Plugin Information Schema tables can be for troubleshooting. You can find even more details/examples at the following location: http://dev.mysql.com/doc/innodb-plugin/1.0/en/innodb-information-schema-examples.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 19:47:41 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:9:{i:0;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:18:"information schema";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:6:"innodb";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:25:"innodb information schema";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:19:"innodb lock monitor";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:13:"innodb plugin";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:12:"INNODB_LOCKS";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:17:"INNODB_LOCK_WAITS";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:10:"INNODB_TRX";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:6420:"

Tracking down InnoDB lock information using the new Information Schema tables provided with the InnoDB plugin has never been easier.

Long story short, the other day I was trying to identify what transaction was holding the lock for a particular UPDATE. The UPDATE would not complete and kept timing out with “Lock wait timeout exceeded; try restarting transaction”.

Of course I checked the output of SHOW ENGINE INNODB STATUS. From that output, I could tell it was the replication thread holding the lock, *since* it was the only other transaction running. But I could not verify this with output, it just had to be the case.

Next I enabled the InnoDB lock monitor, and examined that output too. Here, it showed more detail on the locks being held, however, it only shows the first 10 locks held by the replication thread. In this case, the replication thread was holding more than 2000 record locks from a large, open transaction. Needless to say, the lock I was looking for was not amongst the 10 displayed.

So I enabled InnoDB plugin (just added the following 2 lines to my config file and restarted mysqld):

ignore_builtin_innodb
plugin-load=innodb=ha_innodb_plugin.dll;innodb_trx=ha_innodb_plugin.dll;
innodb_locks=ha_innodb_plugin.dll;innodb_lock_waits=ha_innodb_plugin.dll;
innodb_cmp=ha_innodb_plugin.dll;innodb_cmp_reset=ha_innodb_plugin.dll;
innodb_cmpmem=ha_innodb_plugin.dll;innodb_cmpmem_reset=ha_innodb_plugin.dll

After the above, you’ll notice the following new tables in Information Schema:

mysql> SHOW TABLES IN INFORMATION_SCHEMA LIKE 'INNODB_%';
+-----------------------------------------+
| Tables_in_information_schema (INNODB_%) |
+-----------------------------------------+
| INNODB_CMP_RESET                        |
| INNODB_TRX                              |
| INNODB_CMPMEM_RESET                     |
| INNODB_LOCK_WAITS                       |
| INNODB_CMPMEM                           |
| INNODB_CMP                              |
| INNODB_LOCKS                            |
+-----------------------------------------+
7 rows in set (0.00 sec)

For this particular problem, I wanted to see what was in INNODB_TRX, INNODB_LOCKS, and INNODB_LOCK_WAITS:

mysql> SELECT * FROM INFORMATION_SCHEMA.INNODB_TRX;
+--------+-----------+---------------------+-----------------------+---------------------+------------+---------------------+-----------+
| trx_id | trx_state | trx_started | trx_requested_lock_id | trx_wait_started | trx_weight | trx_mysql_thread_id | trx_query |
+--------+-----------+---------------------+-----------------------+---------------------+------------+---------------------+-----------+
| 124616 | LOCK WAIT | 2010-06-23 13:51:52 | 124616:1301:1572:6 | 2010-06-23 13:51:52 | 2 | 496854 | NULL |
| 123900 | RUNNING | 2010-06-23 09:30:59 | NULL | NULL | 506 | 3 | NULL |
+--------+-----------+---------------------+-----------------------+---------------------+------------+---------------------+-----------+
2 rows in set (0.00 sec)

mysql> SELECT * FROM INFORMATION_SCHEMA.INNODB_LOCKS;
+--------------------+-------------+-----------+-----------+-------------------------+------------+------------+-----------+----------+-----------+
| lock_id | lock_trx_id | lock_mode | lock_type | lock_table | lock_index | lock_space | lock_page | lock_rec | lock_data |
+--------------------+-------------+-----------+-----------+-------------------------+------------+------------+-----------+----------+-----------+
| 124616:1301:1572:6 | 124616 | X | RECORD | `db1`.`t1` | `PRIMARY` | 1301 | 1572 | 6 | 4100 |
| 123900:1301:1572:6 | 123900 | S | RECORD | `db1`.`t1` | `PRIMARY` | 1301 | 1572 | 6 | 4100 |
+--------------------+-------------+-----------+-----------+-------------------------+------------+------------+-----------+----------+-----------+
2 rows in set (0.01 sec)

mysql> SELECT * FROM INFORMATION_SCHEMA.INNODB_LOCK_WAITS;
+-------------------+--------------------+-----------------+--------------------+
| requesting_trx_id | requested_lock_id | blocking_trx_id | blocking_lock_id |
+-------------------+--------------------+-----------------+--------------------+
| 124616 | 124616:1301:1572:6 | 123900 | 123900:1301:1572:6 |
+-------------------+--------------------+-----------------+--------------------+
1 row in set (0.00 sec)

Finally, I had conclusive evidence that it was indeed the replication thread (trx id 123900) holding the lock which was blocking the UPDATE.

I also found the exact row which was being held, and all of the details, such as it was a S (shared) record lock, and the the lock was on the primary key of table db1.t1. It even tells you the lock_space (1301), the lock_page (1572), the lock_rec (6) , and the actual data, lock_data (4100).

The other key piece was the time (or trx_started) for the running transaction, 123900. With that, we knew exactly where to check the binlogs for the exact statement where this all began.

In conclusion, I wish I had started here to begin with, so keep this in mind for the future. And this is just one example of how beneficial these new InnoDB Plugin Information Schema tables can be for troubleshooting. You can find even more details/examples at the following location:

http://dev.mysql.com/doc/innodb-plugin/1.0/en/innodb-information-schema-examples.html


PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:14:"Chris Calender";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:4;a:6:{s:4:"data";s:38:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:49:"Rows and Columns, We Can Definitely Live Together";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:95:"http://www.infobright.org/site/rows_and_columns_we_can_definitely_live_together/#When:19:08:12Z";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:80:"http://www.infobright.org/site/rows_and_columns_we_can_definitely_live_together/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:2760:"In the world of relational database, we are all aware of the benefits that row oriented databases deliver to businesses. They have been designed to support the relational model, have driven the broad adoption of an approachable open language for managing data (SQL), and have been the foundation for high transaction applications including online airline reservation systems, massive online commerce solutions and massively multiplayer online games. Over the course of many years these entrenched row oriented databases have been challenged by the likes of object/relational, NoSQL solutions and even Map/Reduce (though it is certainly not a database). Sometimes row-oriented databases are either overkill or not a suitable fit for the task at hand. NoSQL differs significantly from the row oriented relational database, especially when it comes to dealing with large volumes of unstructured data. As an example it may be useful for rapidly storing and retrieving large volumes of user profile data or applications that don't have heavy transaction processing requirements. One may choose a NoSQL solution for a particular project where it would be considered a best fit. This doesn't mean the company is going to toss out a row oriented transactional database like Oracle or DB2. This is the same case for columnar databases like Infobright. Infobright was designed to be a low footprint, easy to use, self-tuning, high performing and low cost solution for handling analytic style big data and queries.  We never position ourselves as a database for handling heavy and intensive transactional queries with lots of DML (inserts/updates/deletes). Based on the relational model with MySQL compatibility, we at Infobright find ourselves frequently operating side-by-side the transactional system or replacing transactional databases for projects where they simply had trouble satisfying the analytic needs of the business.  In many cases, executing analytic style queries against a transactional database while that database is also handling large volumes of transactional data can bring the system to its knees, causing a great deal of contention and very poor performing queries. Infobright can sit side-by-side these transactional systems, ingesting data at a pretty rapid clip to support the analytic requirements that would have challenged the transactional row-based relational database. This is a perfect fit for Infobright technology. Therefore, when considering a project that involves lots of analytic data and analytic queries,  where you want to avoid contention with your transactional database, Infobright would be happy to sit by side with your transactional system, where rows and columns can definitely live together.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 19:08:12 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:2973:"In the world of relational database, we are all aware of the benefits that row oriented databases deliver to businesses. They have been designed to support the relational model, have driven the broad adoption of an approachable open language for managing data (SQL), and have been the foundation for high transaction applications including online airline reservation systems, massive online commerce solutions and massively multiplayer online games. Over the course of many years these entrenched row oriented databases have been challenged by the likes of object/relational, NoSQL solutions and even Map/Reduce (though it is certainly not a database). Sometimes row-oriented databases are either overkill or not a suitable fit for the task at hand. NoSQL differs significantly from the row oriented relational database, especially when it comes to dealing with large volumes of unstructured data. As an example it may be useful for rapidly storing and retrieving large volumes of user profile data or applications that don't have heavy transaction processing requirements. One may choose a NoSQL solution for a particular project where it would be considered a best fit. This doesn't mean the company is going to toss out a row oriented transactional database like Oracle or DB2. This is the same case for columnar databases like Infobright. Infobright was designed to be a low footprint, easy to use, self-tuning, high performing and low cost solution for handling analytic style big data and queries.  We never position ourselves as a database for handling heavy and intensive transactional queries with lots of DML (inserts/updates/deletes). Based on the relational model with MySQL compatibility, we at Infobright find ourselves frequently operating side-by-side the transactional system or replacing transactional databases for projects where they simply had trouble satisfying the analytic needs of the business.  In many cases, executing analytic style queries against a transactional database while that database is also handling large volumes of transactional data can bring the system to its knees, causing a great deal of contention and very poor performing queries. Infobright can sit side-by-side these transactional systems, ingesting data at a pretty rapid clip to support the analytic requirements that would have challenged the transactional row-based relational database. This is a perfect fit for Infobright technology. Therefore, when considering a project that involves lots of analytic data and analytic queries,  where you want to avoid contention with your transactional database, Infobright would be happy to sit by side with your transactional system, where rows and columns can definitely live together.
PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:7:"Dirksen";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:5;a:6:{s:4:"data";s:48:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:89:"High availability for MySQL on Amazon EC2 – Part 2 – Setting up the initial instances";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:43:"http://www.mysqlperformanceblog.com/?p=3094";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:139:"http://www.mysqlperformanceblog.com/2010/06/29/high-availability-for-mysql-on-amazon-ec2-%E2%80%93-part-2-setting-up-the-initial-instances/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:13553:"This post is the second of a series that started here. The first step to build the HA solution is to create two working instances, configure them to be EBS based and create a security group for them. A third instance, the client, will be discussed in part 7. Since this will be a proof of concept, I'll be using m1.small type instances while normally, the mysql host would be much larger. Using another type is trivial. I will assume you are using the command line api tools, on Ubuntu, install "ec2-api-tools". The use of these tools simplifies the expression of the command compared to the web based console. Create the security group The instances involved in the MySQL HA setup will need to be inside the same security group for networking purposes and the help identify them. To create a security simply run this command: PLAIN TEXT CODE: yves@yves-laptop:~$ export EC2_CERT=cert-yves.pem yves@yves-laptop:~$ export EC2_PRIVATE_KEY=pk-yves.pem yves@yves-laptop:~$ ec2-add-group hamysql -d 'nodes for HA MySQL solution' GROUP   hamysql nodes for HA MySQL solution From now, I'll always assume the EC2_CERT and EC2_PRIVATE_KEY environment variables are setup in your shell. Next, we need to authorize some communications for the security group. I'll authorize 3306/tcp (MySQL) from hamysql, 694/udp (Heartbeat) from hamysql and 22 (SSH) from everywhere. You can be more restrictive for SSH if you want to. PLAIN TEXT CODE: yves@yves-laptop:~$ ec2-authorize hamysql -P tcp -p 3306  -o hamysql -u 834362721059 yves@yves-laptop:~$ ec2-authorize hamysql -P udp -p 694  -o hamysql -u 834362721059 yves@yves-laptop:~$ ec2-authorize hamysql -P tcp -p 22 -s 0.0.0.0/0 yves@yves-laptop:~$  ec2-describe-group hamysql GROUP   834362721059    hamysql nodes for HA MySQL solution PERMISSION      834362721059    hamysql ALLOWS  tcp     3306    3306    FROM    USER    834362721059    GRPNAME hamysql PERMISSION      834362721059    hamysql ALLOWS  udp     694     694     FROM    USER    834362721059    GRPNAME hamysql PERMISSION      834362721059    hamysql ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0 Launch the instances Now we can start creating our instances. Since this is only a proof of concept, I'll built 2 m1.small instances, fell free to use other types. At the time I wrote this, the following AMI seems ok. PLAIN TEXT CODE: yves@yves-laptop:~$  ec2-describe-images ami-1cdf3775 IMAGE   ami-1cdf3775    099720109477/ubuntu-images-testing/ubuntu-lucid-daily-i386-server-20100618      099720109477    available       public          i386    machine aki-aca44cc5         instance-store So, lauching 2 of these, PLAIN TEXT CODE: yves@yves-laptop:~$  ec2-run-instances ami-1cdf3775 -n 2 -g hamysql -t m1.small -k yves-key RESERVATION     r-a29c31c9      834362721059    hamysql INSTANCE        i-a23a21c9      ami-1cdf3775                    pending yves-key        0               m1.small        2010-06-18T20:11:14+0000        us-east-1c      aki-aca44cc5         monitoring-disabled                                      instance-store INSTANCE        i-a03a21cb      ami-1cdf3775                    pending yves-key        1               m1.small        2010-06-18T20:11:14+0000        us-east-1c      aki-aca44cc5         monitoring-disabled yves@yves-laptop:~$  ec2-describe-instances RESERVATION     r-a29c31c9      834362721059    hamysql INSTANCE        i-a23a21c9      ami-1cdf3775    ec2-174-129-89-188.compute-1.amazonaws.com      domU-12-31-39-02-BD-C5.compute-1.internal       running yves-key        0               m1.small      2010-06-18T20:11:14+0000        us-east-1c      aki-aca44cc5                    monitoring-disabled     174.129.89.188  10.248.194.51                   instance-store INSTANCE        i-a03a21cb      ami-1cdf3775    ec2-174-129-187-170.compute-1.amazonaws.com     domU-12-31-39-03-A4-62.compute-1.internal       running yves-key        1               m1.small      2010-06-18T20:11:14+0000        us-east-1c      aki-aca44cc5                    monitoring-disabled     174.129.187.170 10.249.167.144                  instance-store I don't know about you but I don't like multi-lines output so I wrote a small filter script to on one line the parameters I need separated by a delimiter. PLAIN TEXT CODE: yves@yves-laptop:~$  cat  filtre_instances.pl #!/usr/bin/perl   $SecGroup = ''; $IPAdd = ''; $Instance_ID = '';   while (<STDIN>) {         chomp $_;           #print "Processing: $_\n";           @fields = split /\t/, $_;           if (/^RESERVATION/)     {                 $SecGroup = $fields[3];         }         if (/^INSTANCE/) {                 $IPAdd = $fields[17];                 $STORE = $fields[20];                 $Instance_ID= $fields[1];                 $AMI_ID= $fields[2];                 $PUBDNS = $fields[3];                 $STATUS = $fields[5];                 $START = $fields[10];                 print "$SecGroup|$IPAdd|$Instance_ID|$AMI_ID|$PUBDNS|$STATUS|$START|$STORE \n"         } } and now we have PLAIN TEXT CODE: yves@yves-laptop:~$ ec2-describe-instances | ./filtre_instances.pl | grep hamysql hamysql|10.248.194.51|i-a23a21c9|ami-1cdf3775|ec2-174-129-89-188.compute-1.amazonaws.com|running|2010-06-18T20:11:14+0000|instance-store hamysql|10.249.167.144|i-a03a21cb|ami-1cdf3775|ec2-174-129-187-170.compute-1.amazonaws.com|running|2010-06-18T20:11:14+0000|instance-store which is to my opinion easier to manipulate. Configuring Heartbeat Now, let's configure Heartbeat. The first thing to is to set up the hostname on both host. Heartbeat identifies the host on which it is running by its hostname so that's mandatory step. First host: PLAIN TEXT CODE: yves@yves-laptop:~$ ssh -i ~/.ssh/yves-key.pem ubuntu@ec2-174-129-89-188.compute-1.amazonaws.com ubuntu@domU-12-31-39-07-C8-32:~$ sudo su - ubuntu@domU-12-31-39-07-C8-32:~# hostname monitor Second host: PLAIN TEXT CODE: yves@yves-laptop:~$ ssh -i ~/.ssh/yves-key.pem ubuntu@ec2-174-129-89-188.compute-1.amazonaws.com ubuntu@domU-12-31-38-04-E5-E4:~$ sudo su - ubuntu@domU-12-31-38-04-E5-E4:~# hostname hamysql We don't really need to set /etc/hostname since it is overwritten when the instance is started, even when using EBS based AMI. The next step is to install Heartbeat and Pacemaker on both host, with Ubuntu 10.04, it is very straightforward: PLAIN TEXT CODE: root@monitor:~# apt-get install heartbeat pacemaker   and   root@hamysql:~# apt-get install heartbeat pacemaker Then we can proceed and configure Heartbeat, Pacemaker will come later. Heartbeat needs 2 configuration files, /etc/ha.d/authkeys for cluster authentication and /etc/ha.d/ha.cf which is the configuration file per say. On both host, the chosen key in the authkeys file must be identical and good way to generate unique one is to run "date | md5sum" and grab a substring from the output. PLAIN TEXT CODE: root@monitor:/etc/ha.d# cat authkeys auth 1 1 sha1 c97f2bb4b5ae90f149dc314ed Also don't forget to restrict the access rights on the file like: PLAIN TEXT CODE: root@monitor:/etc/ha.d# chmod 600 authkeys For the /etc/ha.d/ha.cf file, since EC2 does not support neither broadcast or multicast within the security group, we need to use unicast (ucast) so both files will not be identical. The ucast entry on one host will contain the IP address on the internal network of the other host. On the monitor host, we will have: PLAIN TEXT CODE: root@monitor:/etc/ha.d# cat ha.cf autojoin none ucast eth0 10.249.167.144 warntime 5 deadtime 15 initdead 60 keepalive 2 crm respawn node monitor node hamysql and on the hamysql host: PLAIN TEXT CODE: root@hamysql:/etc/ha.d# cat ha.cf autojoin none ucast eth0 10.248.194.51 warntime 5 deadtime 15 initdead 60 keepalive 2 crm respawn node monitor node hamysql Let's review briefly the configuration file. First we have setup "autojoin none" that means no host not listed explicitely in the configuration file can join the cluster so we know we have at most 2 members, "monitor" and "hamysql". Next is the ucast communication channel to reach the other node and the timing parameters. "warntime" is a soft timeout in second that logs the other node is later while "deadtime" is the hard limit after which heartbeat will the consider the other node dead and start actions to restore the service. The "initdead" is just a startup delay to allow host to fully boot before attempting actions and "crm respawn" starts the Pacemaker resources manager. Finally, we have the two "node" declarations" for the cluster members. So we are done with the configure, time to try if it works. On both hosts, run: PLAIN TEXT CODE: service heartbeat start and if everything is right, after at most a minute, you should be able to see both heartbeat processes chatting over the network PLAIN TEXT CODE: root@monitor:~# tcpdump -i eth0 port 694 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 20:38:36.536302 IP domU-12-31-38-04-E5-E4.compute-1.internal.57802> domU-12-31-39-07-C8-32.compute-1.internal.694: UDP, length 211 20:38:36.928860 IP domU-12-31-39-07-C8-32.compute-1.internal.34058> domU-12-31-38-04-E5-E4.compute-1.internal.694: UDP, length 212 20:38:38.580245 IP domU-12-31-38-04-E5-E4.compute-1.internal.57802> domU-12-31-39-07-C8-32.compute-1.internal.694: UDP, length 211 20:38:38.938814 IP domU-12-31-39-07-C8-32.compute-1.internal.34058> domU-12-31-38-04-E5-E4.compute-1.internal.694: UDP, length 212 We can also use the "crm" tool to query the cluster status. PLAIN TEXT CODE: root@monitor:~# crm status ============ Last updated: Tue Jun 29 13:56:04 2010 Stack: Heartbeat Current DC: monitor (504f45ea-7aee-4fa5-b0ee-a5ac07975ce4) - partition with quorum Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd 2 Nodes configured, unknown expected votes 1 Resources configured. ============   Online: [ monitor hamysql ] Install MySQL For the sake of simplicity, we will just install the MySQL version in the Ubuntu repository by doing: PLAIN TEXT CODE: root@hamysql:~# apt-get install mysql-server-5.1 The package install MySQL has an automatic startup script controlled by init (new to lucid). That's fine, I will surprise you but Pacemaker will not manager MySQL, just the host running it. I'll also skip the raid configuration of multiple EBS volumes since it is not the main purpose of this blog series. EBS based AMI Others have produce excellent article on how to create EBS based AMI, I will not reinvent the wheel. I followed this one: http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html Upcoming in part 3, the configuration of the HA resources. Entry posted by Yves Trudeau | No comment Add to: | | | | ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 15:23:45 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:2:{i:0;a:5:{s:4:"data";s:17:"High Availability";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:26000:"

This post is the second of a series that started here.

The first step to build the HA solution is to create two working instances, configure them to be EBS based and create a security group for them. A third instance, the client, will be discussed in part 7. Since this will be a proof of concept, I'll be using m1.small type instances while normally, the mysql host would be much larger. Using another type is trivial. I will assume you are using the command line api tools, on Ubuntu, install "ec2-api-tools". The use of these tools simplifies the expression of the command compared to the web based console.

Create the security group

The instances involved in the MySQL HA setup will need to be inside the same security group for networking purposes and the help identify them. To create a security simply run this command:

PLAIN TEXT
CODE:
  1. yves@yves-laptop:~$ export EC2_CERT=cert-yves.pem
  2. yves@yves-laptop:~$ export EC2_PRIVATE_KEY=pk-yves.pem
  3. yves@yves-laptop:~$ ec2-add-group hamysql -d 'nodes for HA MySQL solution'
  4. GROUP   hamysql nodes for HA MySQL solution

From now, I'll always assume the EC2_CERT and EC2_PRIVATE_KEY environment variables are setup in your shell. Next, we need to authorize some communications for the security group. I'll authorize 3306/tcp (MySQL) from hamysql, 694/udp (Heartbeat) from hamysql and 22 (SSH) from everywhere. You can be more restrictive for SSH if you want to.

PLAIN TEXT
CODE:
  1. yves@yves-laptop:~$ ec2-authorize hamysql -P tcp -p 3306  -o hamysql -u 834362721059
  2. yves@yves-laptop:~$ ec2-authorize hamysql -P udp -p 694  -o hamysql -u 834362721059
  3. yves@yves-laptop:~$ ec2-authorize hamysql -P tcp -p 22 -s 0.0.0.0/0
  4. yves@yves-laptop:~$  ec2-describe-group hamysql
  5. GROUP   834362721059    hamysql nodes for HA MySQL solution
  6. PERMISSION      834362721059    hamysql ALLOWS  tcp     3306    3306    FROM    USER    834362721059    GRPNAME hamysql
  7. PERMISSION      834362721059    hamysql ALLOWS  udp     694     694     FROM    USER    834362721059    GRPNAME hamysql
  8. PERMISSION      834362721059    hamysql ALLOWS  tcp     22      22      FROM    CIDR    0.0.0.0/0

Launch the instances

Now we can start creating our instances. Since this is only a proof of concept, I'll built 2 m1.small instances, fell free to use other types. At the time I wrote this, the following AMI seems ok.

PLAIN TEXT
CODE:
  1. yves@yves-laptop:~$  ec2-describe-images ami-1cdf3775
  2. IMAGE   ami-1cdf3775    099720109477/ubuntu-images-testing/ubuntu-lucid-daily-i386-server-20100618      099720109477    available       public          i386    machine aki-aca44cc5         instance-store

So, lauching 2 of these,

PLAIN TEXT
CODE:
  1. yves@yves-laptop:~$  ec2-run-instances ami-1cdf3775 -n 2 -g hamysql -t m1.small -k yves-key
  2. RESERVATION     r-a29c31c9      834362721059    hamysql
  3. INSTANCE        i-a23a21c9      ami-1cdf3775                    pending yves-key        0               m1.small        2010-06-18T20:11:14+0000        us-east-1c      aki-aca44cc5         monitoring-disabled                                      instance-store
  4. INSTANCE        i-a03a21cb      ami-1cdf3775                    pending yves-key        1               m1.small        2010-06-18T20:11:14+0000        us-east-1c      aki-aca44cc5         monitoring-disabled
  5. yves@yves-laptop:~$  ec2-describe-instances
  6. RESERVATION     r-a29c31c9      834362721059    hamysql
  7. INSTANCE        i-a23a21c9      ami-1cdf3775    ec2-174-129-89-188.compute-1.amazonaws.com      domU-12-31-39-02-BD-C5.compute-1.internal       running yves-key        0               m1.small      2010-06-18T20:11:14+0000        us-east-1c      aki-aca44cc5                    monitoring-disabled     174.129.89.188  10.248.194.51                   instance-store
  8. INSTANCE        i-a03a21cb      ami-1cdf3775    ec2-174-129-187-170.compute-1.amazonaws.com     domU-12-31-39-03-A4-62.compute-1.internal       running yves-key        1               m1.small      2010-06-18T20:11:14+0000        us-east-1c      aki-aca44cc5                    monitoring-disabled     174.129.187.170 10.249.167.144                  instance-store

I don't know about you but I don't like multi-lines output so I wrote a small filter script to on one line the parameters I need separated by a delimiter.

PLAIN TEXT
CODE:
  1. yves@yves-laptop:~$  cat  filtre_instances.pl
  2. #!/usr/bin/perl
  3.  
  4. $SecGroup = '';
  5. $IPAdd = '';
  6. $Instance_ID = '';
  7.  
  8. while (<STDIN>) {
  9.         chomp $_;
  10.  
  11.         #print "Processing: $_\n";
  12.  
  13.         @fields = split /\t/, $_;
  14.  
  15.         if (/^RESERVATION/)     {
  16.                 $SecGroup = $fields[3];
  17.         }
  18.         if (/^INSTANCE/) {
  19.                 $IPAdd = $fields[17];
  20.                 $STORE = $fields[20];
  21.                 $Instance_ID= $fields[1];
  22.                 $AMI_ID= $fields[2];
  23.                 $PUBDNS = $fields[3];
  24.                 $STATUS = $fields[5];
  25.                 $START = $fields[10];
  26.                 print "$SecGroup|$IPAdd|$Instance_ID|$AMI_ID|$PUBDNS|$STATUS|$START|$STORE \n"
  27.         }
  28. }

and now we have

PLAIN TEXT
CODE:
  1. yves@yves-laptop:~$ ec2-describe-instances | ./filtre_instances.pl | grep hamysql
  2. hamysql|10.248.194.51|i-a23a21c9|ami-1cdf3775|ec2-174-129-89-188.compute-1.amazonaws.com|running|2010-06-18T20:11:14+0000|instance-store
  3. hamysql|10.249.167.144|i-a03a21cb|ami-1cdf3775|ec2-174-129-187-170.compute-1.amazonaws.com|running|2010-06-18T20:11:14+0000|instance-store

which is to my opinion easier to manipulate.

Configuring Heartbeat

Now, let's configure Heartbeat. The first thing to is to set up the hostname on both host. Heartbeat identifies the host on which it is running by its hostname so that's mandatory step.

First host:

PLAIN TEXT
CODE:
  1. yves@yves-laptop:~$ ssh -i ~/.ssh/yves-key.pem ubuntu@ec2-174-129-89-188.compute-1.amazonaws.com
  2. ubuntu@domU-12-31-39-07-C8-32:~$ sudo su -
  3. ubuntu@domU-12-31-39-07-C8-32:~# hostname monitor

Second host:

PLAIN TEXT
CODE:
  1. yves@yves-laptop:~$ ssh -i ~/.ssh/yves-key.pem ubuntu@ec2-174-129-89-188.compute-1.amazonaws.com
  2. ubuntu@domU-12-31-38-04-E5-E4:~$ sudo su -
  3. ubuntu@domU-12-31-38-04-E5-E4:~# hostname hamysql

We don't really need to set /etc/hostname since it is overwritten when the instance is started, even when using EBS based AMI. The next step is to install Heartbeat and Pacemaker on both host, with Ubuntu 10.04, it is very straightforward:

PLAIN TEXT
CODE:
  1. root@monitor:~# apt-get install heartbeat pacemaker
  2.  
  3. and
  4.  
  5. root@hamysql:~# apt-get install heartbeat pacemaker

Then we can proceed and configure Heartbeat, Pacemaker will come later. Heartbeat needs 2 configuration files, /etc/ha.d/authkeys for cluster authentication and /etc/ha.d/ha.cf which is the configuration file per say. On both host, the chosen key in the authkeys file must be identical and good way to generate unique one is to run "date | md5sum" and grab a substring from the output.

PLAIN TEXT
CODE:
  1. root@monitor:/etc/ha.d# cat authkeys
  2. auth 1
  3. 1 sha1 c97f2bb4b5ae90f149dc314ed

Also don't forget to restrict the access rights on the file like:

PLAIN TEXT
CODE:
  1. root@monitor:/etc/ha.d# chmod 600 authkeys

For the /etc/ha.d/ha.cf file, since EC2 does not support neither broadcast or multicast within the security group, we need to use unicast (ucast) so both files will not be identical. The ucast entry on one host will contain the IP address on the internal network of the other host. On the monitor host, we will have:

PLAIN TEXT
CODE:
  1. root@monitor:/etc/ha.d# cat ha.cf
  2. autojoin none
  3. ucast eth0 10.249.167.144
  4. warntime 5
  5. deadtime 15
  6. initdead 60
  7. keepalive 2
  8. crm respawn
  9. node monitor
  10. node hamysql

and on the hamysql host:

PLAIN TEXT
CODE:
  1. root@hamysql:/etc/ha.d# cat ha.cf
  2. autojoin none
  3. ucast eth0 10.248.194.51
  4. warntime 5
  5. deadtime 15
  6. initdead 60
  7. keepalive 2
  8. crm respawn
  9. node monitor
  10. node hamysql

Let's review briefly the configuration file. First we have setup "autojoin none" that means no host not listed explicitely in the configuration file can join the cluster so we know we have at most 2 members, "monitor" and "hamysql". Next is the ucast communication channel to reach the other node and the timing parameters. "warntime" is a soft timeout in second that logs the other node is later while "deadtime" is the hard limit after which heartbeat will the consider the other node dead and start actions to restore the service. The "initdead" is just a startup delay to allow host to fully boot before attempting actions and "crm respawn" starts the Pacemaker resources manager. Finally, we have the two "node" declarations" for the cluster members.

So we are done with the configure, time to try if it works. On both hosts, run:

PLAIN TEXT
CODE:
  1. service heartbeat start

and if everything is right, after at most a minute, you should be able to see both heartbeat processes chatting over the network

PLAIN TEXT
CODE:
  1. root@monitor:~# tcpdump -i eth0 port 694
  2. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  3. listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
  4. 20:38:36.536302 IP domU-12-31-38-04-E5-E4.compute-1.internal.57802> domU-12-31-39-07-C8-32.compute-1.internal.694: UDP, length 211
  5. 20:38:36.928860 IP domU-12-31-39-07-C8-32.compute-1.internal.34058> domU-12-31-38-04-E5-E4.compute-1.internal.694: UDP, length 212
  6. 20:38:38.580245 IP domU-12-31-38-04-E5-E4.compute-1.internal.57802> domU-12-31-39-07-C8-32.compute-1.internal.694: UDP, length 211
  7. 20:38:38.938814 IP domU-12-31-39-07-C8-32.compute-1.internal.34058> domU-12-31-38-04-E5-E4.compute-1.internal.694: UDP, length 212

We can also use the "crm" tool to query the cluster status.

PLAIN TEXT
CODE:
  1. root@monitor:~# crm status
  2. ============
  3. Last updated: Tue Jun 29 13:56:04 2010
  4. Stack: Heartbeat
  5. Current DC: monitor (504f45ea-7aee-4fa5-b0ee-a5ac07975ce4) - partition with quorum
  6. Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
  7. 2 Nodes configured, unknown expected votes
  8. 1 Resources configured.
  9. ============
  10.  
  11. Online: [ monitor hamysql ]

Install MySQL

For the sake of simplicity, we will just install the MySQL version in the Ubuntu repository by doing:

PLAIN TEXT
CODE:
  1. root@hamysql:~# apt-get install mysql-server-5.1

The package install MySQL has an automatic startup script controlled by init (new to lucid). That's fine, I will surprise you but Pacemaker will not manager MySQL, just the host running it. I'll also skip the raid configuration of multiple EBS volumes since it is not the main purpose of this blog series.

EBS based AMI

Others have produce excellent article on how to create EBS based AMI, I will not reinvent the wheel. I followed this one: http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html

Upcoming in part 3, the configuration of the HA resources.

Entry posted by Yves Trudeau | No comment

Add to: delicious | digg | reddit | netscape | Google Bookmarks


PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:12:"Yves Trudeau";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:6;a:6:{s:4:"data";s:78:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:43:"There’s a European OpenSQL Camp coming up";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:33:"http://www.xaprb.com/blog/?p=1905";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:78:"http://www.xaprb.com/blog/2010/06/29/theres-a-european-opensql-camp-coming-up/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:577:"In addition to the Boston edition, there’s an OpenSQL Camp at the same time and place as FrOSCon mid-August in Germany. The call for papers is open until July 11th. As always, the conference is about all kinds of open-source databases: MySQL and PostgreSQL are only two of the obvious ones; MongoDB and Cassandra featured prominently at the last one I attended, and SQLite was well represented at the first one. Related posts:OpenSQL Camp Boston 2010OpenSQL Camp events in 2009Recap of Portland OpenSQL Camp 2009The history of OpenSQL CampOpenSQL Camp develops further";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 12:32:53 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:8:{i:0;a:5:{s:4:"data";s:11:"Conferences";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:12:"OpenSQL Camp";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:10:"PostgreSQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:3:"SQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:9:"cassandra";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:14:"Giuseppe Maxia";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:7:"MongoDB";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:6:"SQLite";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1664:"

In addition to the Boston edition, there’s an OpenSQL Camp at the same time and place as FrOSCon mid-August in Germany. The call for papers is open until July 11th. As always, the conference is about all kinds of open-source databases: MySQL and PostgreSQL are only two of the obvious ones; MongoDB and Cassandra featured prominently at the last one I attended, and SQLite was well represented at the first one.

Related posts:

  1. OpenSQL Camp Boston 2010
  2. OpenSQL Camp events in 2009
  3. Recap of Portland OpenSQL Camp 2009
  4. The history of OpenSQL Camp
  5. OpenSQL Camp develops further


PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:22:"Baron Schwartz (xaprb)";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:7;a:6:{s:4:"data";s:53:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:38:"Drizzle @ Velocity (seemed to go well)";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:40:"http://www.flamingspork.com/blog/?p=2047";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:79:"http://www.flamingspork.com/blog/2010/06/29/drizzle-velocity-seemed-to-go-well/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:539:"Monty’s talk at Velocity 2010 seemed to go down really well (at least from reading the agile admin entry on Drizzle). There are a few great bits from this article that just made me laugh: “Oracle’s “run Java within the database” is an example of totally retarded functionality whose main job is to ruin your life” Love it that we’re managing to get the message out. Share this on Facebook Tweet This! Share this on del.icio.us Digg this! Post on Google Buzz ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 11:24:10 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:3:{i:0;a:5:{s:4:"data";s:7:"drizzle";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:10:"conference";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:8:"velocity";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:2038:"

Monty’s talk at Velocity 2010 seemed to go down really well (at least from reading the agile admin entry on Drizzle). There are a few great bits from this article that just made me laugh:

“Oracle’s “run Java within the database” is an example of totally retarded functionality whose main job is to ruin your life”

Love it that we’re managing to get the message out.


PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Stewart Smith";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:8;a:6:{s:4:"data";s:58:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:57:"Time for an UPDATE using the Maria Storage Engine then...";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:70:"tag:blogger.com,1999:blog-9144505959002328789.post-7165290207366716312";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:88:"http://karlssonondatabases.blogspot.com/2010/06/time-for-update-using-maria-storage.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:1858:"Again, another simple test. Same basic tuning as yesterday, and the table schema is the same:CREATE TABLE `t1` ( `c1` int(11) NOT NULL AUTO_INCREMENT, `c2` char(100) DEFAULT NULL, PRIMARY KEY (`c1`));The table is again filled with 1.3 million rows, but this time I'm doing an UPDATE. The update is again a simple primary key update:UPDATE t1 SET c2 = CONCAT('xxx', RAND(), 'yyy') WHERE c1 = <random value 1 - 1000000>;I run this on the Maria, InnoDB and MyISAM engines. The issues with the MyISAM and Maria engines here is that they lack row level locking. In MariaDB 5.1.47, the InnoDB version is 1.0.6, so it is more scalable than what it used to be. The testbench is not an incredibly hot machiine, just a 4 core AMD box.I run the test in some different configurations, using a single thread, using 10 threads and using 100 threads. Here we can see that InnoDB Row-level locking really hlps. With InnoDB, performance was on par with Maria with 1 thread, MyISAM lagging behind (yes, here Maria was faster than MyISAM, it might be some MyISAM tuning that I missed, beyond basic key_buffer_size and all that).Moving to 10 threads, InnoDB could now perform about twice as many operations per second, whereas Maris throughput increased by aout 15%, MyISAM stayed the same old slow. Going to 100 threads, again InnoDB nearly doubled the number of ops, Maria abot 15% up, and MyISAM still about the same, actually slightly slower both at 10 and 100 threads.Is this a scientific test? Nope, hardly. Does it reflect something that might be happening in a system in the real world? Yes, I'd say so, given that there would be other things going on also.All in all, I think the Maria Engine has improved a bit, and may be an alternative to MyiSAM, if it wasn't for the fact that InnoDB really kicks them both in their rearward facing bodily regions./Karlsson";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 11:21:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:11:"performance";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:6:"innodb";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:5:"maria";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:2524:"Again, another simple test. Same basic tuning as yesterday, and the table schema is the same:
CREATE TABLE `t1` (
`c1` int(11) NOT NULL AUTO_INCREMENT,
`c2` char(100) DEFAULT NULL,
PRIMARY KEY (`c1`)
);
The table is again filled with 1.3 million rows, but this time I'm doing an UPDATE. The update is again a simple primary key update:
UPDATE t1 SET c2 = CONCAT('xxx', RAND(), 'yyy') WHERE c1 = <random value 1 - 1000000>;
I run this on the Maria, InnoDB and MyISAM engines. The issues with the MyISAM and Maria engines here is that they lack row level locking. In MariaDB 5.1.47, the InnoDB version is 1.0.6, so it is more scalable than what it used to be. The testbench is not an incredibly hot machiine, just a 4 core AMD box.

I run the test in some different configurations, using a single thread, using 10 threads and using 100 threads. Here we can see that InnoDB Row-level locking really hlps. With InnoDB, performance was on par with Maria with 1 thread, MyISAM lagging behind (yes, here Maria was faster than MyISAM, it might be some MyISAM tuning that I missed, beyond basic key_buffer_size and all that).

Moving to 10 threads, InnoDB could now perform about twice as many operations per second, whereas Maris throughput increased by aout 15%, MyISAM stayed the same old slow. Going to 100 threads, again InnoDB nearly doubled the number of ops, Maria abot 15% up, and MyISAM still about the same, actually slightly slower both at 10 and 100 threads.

Is this a scientific test? Nope, hardly. Does it reflect something that might be happening in a system in the real world? Yes, I'd say so, given that there would be other things going on also.

All in all, I think the Maria Engine has improved a bit, and may be an alternative to MyiSAM, if it wasn't for the fact that InnoDB really kicks them both in their rearward facing bodily regions.

/Karlsson


PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:15:"Anders Karlsson";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:9;a:6:{s:4:"data";s:53:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:48:"OpenSQLCamp EU 2010 - Last days to submit a talk";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:59:"tag:blogger.com,1999:blog-16959946.post-2676292190971872953";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:84:"http://datacharmer.blogspot.com/2010/06/opensqlcamp-eu-2010-last-days-to-submit.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:497:" There is still time and hope if you want to present at the OpenSQLCamp 2010, European edition. The Call for Participation is open until July 11th. Thanks for the ones who have submitted proposals. A few caveats: if your proposal does not include a description, it will not be accepted. Therefore, wannabe speakers, please check your proposals, and make them as good as you can!We want talk about all open source database. Not only MySQL. Open source database fans, wake up, and submit a proposal!";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 08:50:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:3:{i:0;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:11:"opensqlcamp";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:7:"froscon";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1359:"
There is still time and hope if you want to present at the OpenSQLCamp 2010, European edition. The Call for Participation is open until July 11th.
Thanks for the ones who have submitted proposals.
A few caveats: if your proposal does not include a description, it will not be accepted. Therefore, wannabe speakers, please check your proposals, and make them as good as you can!
We want talk about all open source database. Not only MySQL.
Open source database fans, wake up, and submit a proposal!

PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:14:"Giuseppe Maxia";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:10;a:6:{s:4:"data";s:58:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:33:"mysql hack - altering huge tables";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:59:"tag:blogger.com,1999:blog-13606853.post-2200846088859946028";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:73:"http://jayant7k.blogspot.com/2010/06/mysql-hack-altering-huge-tables.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:409:"You have a huge mysql table - maybe 100 GB. And you need to run alter on it - to either add an index, drop an index, add a column or drop a column. If you run the simple mysql "alter table" command, you will end up spending ages to bring the table back into production. Here is a simple hack to get the thing done. The benefit of the hack is that the alter runs quite fast. But since this is a hack";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 07:01:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:6:"MYISAM";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:11:"alter table";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:4:"hack";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:622:"You have a huge mysql table - maybe 100 GB. And you need to run alter on it - to either add an index, drop an index, add a column or drop a column. If you run the simple mysql "alter table" command, you will end up spending ages to bring the table back into production. Here is a simple hack to get the thing done. The benefit of the hack is that the alter runs quite fast. But since this is a hack
PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:12:"Jayant Kumar";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:11;a:6:{s:4:"data";s:38:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:18:"Q4M 0.9.4 released";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:73:"http://developer.cybozu.co.jp/kazuho/2010/06/q4m-094-release.html?lang=en";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:73:"http://developer.cybozu.co.jp/kazuho/2010/06/q4m-094-release.html?lang=en";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:473:"I have just uploaded Q4M (Queue for MySQL) 0.9.4 to q4m.31tools.com.  There has been no bug fixes since 0.9.3, the only change is the newly added function queue_compact(table_name) that can be used to trigger table compaction manually. If you were looking for a way to control queue compaction timing, it would be worth considering upgrading to 0.9.4. For more information of what compaction is, please refer to my last entry on Q4M describing concurrent compaction.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 05:37:20 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:927:"

I have just uploaded Q4M (Queue for MySQL) 0.9.4 to q4m.31tools.com.  There has been no bug fixes since 0.9.3, the only change is the newly added function queue_compact(table_name) that can be used to trigger table compaction manually.

If you were looking for a way to control queue compaction timing, it would be worth considering upgrading to 0.9.4.

For more information of what compaction is, please refer to my last entry on Q4M describing concurrent compaction.


PlanetMySQL Voting: Vote UP / Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:10:"Kazuho Oku";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:12;a:6:{s:4:"data";s:78:" ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:36:"ENUM now works properly (in Drizzle)";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:40:"http://www.flamingspork.com/blog/?p=2041";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:79:"http://www.flamingspork.com/blog/2010/06/29/enum-now-works-properly-in-drizzle/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:3642:"Over at the Drizzle blog, the recent 2010-06-07 tarball was announced. This tarball release has my fixes for the ENUM type, so that it now works as it should. I was quite amazed that such a small block of code could have so many bugs! One of the most interesting was the documented limit we inherited from MySQL (see the MySQL Docs on ENUM) of a maximum of 65,535 elements for an ENUM column. This all started out from a quite innocent comment of Jay‘s in a code review for adding support for the ENUM data type to the embedded_innodb engine. It was all pretty innocent… saying that I should use a constant instead of the magic 0×10000 number as a limit on an assert for sanity of values getting passed to the engine. Seeing as there wasn’t a constant already in the code for that (surprise number 1), I said I’d fix it properly in a separate patch (creating a bug for it so it wouldn’t get lost) and the code went in. So, now, a few weeks after that, I got around to dealing with that bug (because hey, this was going to be an easy fix that’ll give me a nice sense of accomplishment). A quick look in the Field_enum code raised my suspicions of bugs… I initially wondered if we’d get any error message if a StorageEngine returned a table definition that had too many ENUM elements (for example, 70,000). So, I added a table to the tableprototester plugin (a simple dummy engine that is loaded for testing the parsing of specially constructed table messages) that had 70,000 elements for a single ENUM column. It didn’t throw an error. Darn. It did, however, have an incredibly large result for SHOW CREATE TABLE. Often with bugs like this I may try to see if the problem is something inherited from MySQL. I’ll often file a bug with MySQL as well if that’s the case. If I can, I’ll sometimes attach the associated patch from Drizzle that fixes the bug, sometimes with a patch directly for and tested on MySQL (if it’s not going to take me too long). If these patches are ever applied is a whole other thing – and sometimes you get things like “each engine is meant to have auto_increment behave differently!” – which doesn’t inspire confidence. But anyway, the MySQL limit is somewhere between 10850 and 10900. This is not at all what’s documented. I’ve filed the appropriate bug (Bug #54194) with reproducible test case and the bit of problematic code. It turns out that this is (yet another) limit of the FRM file. The limit is “about 64k FRM”. The bit of code in MySQL that was doing the checking for the ENUM limit was this: /* Hack to avoid bugs with small static rows in MySQL */ reclength=max(file->min_record_length(table_options),reclength); if (info_length+(ulong) create_fields.elements*FCOMP+288+ n_length+int_length+com_length > 65535L || int_count > 255) { my_message(ER_TOO_MANY_FIELDS, ER(ER_TOO_MANY_FIELDS), MYF(0)); DBUG_RETURN(1); } So it’s no surprise to anyone how this specific limit (the number of elements in an ENUM) got missed when I converted Drizzle from using an FRM over to a protobuf based structure. So a bunch of other cleanup later, a whole lot of extra testing and I can pretty confidently state that the ENUM type in Drizzle does work exactly how you think it would. Either way, if you’re getting anywhere near 10,000 choices for an ENUM column you have no doubt already lost. Share this on Facebook Tweet This! Share this on del.icio.us Digg this! Post on Google Buzz ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 03:19:04 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:8:{i:0;a:5:{s:4:"data";s:7:"General";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:7:"drizzle";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:3:"bug";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:4:"enum";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:3:"frm";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:6:"limits";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:8:"protobuf";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:5661:"

Over at the Drizzle blog, the recent 2010-06-07 tarball was announced. This tarball release has my fixes for the ENUM type, so that it now works as it should. I was quite amazed that such a small block of code could have so many bugs! One of the most interesting was the documented limit we inherited from MySQL (see the MySQL Docs on ENUM) of a maximum of 65,535 elements for an ENUM column.

This all started out from a quite innocent comment of Jay‘s in a code review for adding support for the ENUM data type to the embedded_innodb engine. It was all pretty innocent… saying that I should use a constant instead of the magic 0×10000 number as a limit on an assert for sanity of values getting passed to the engine. Seeing as there wasn’t a constant already in the code for that (surprise number 1), I said I’d fix it properly in a separate patch (creating a bug for it so it wouldn’t get lost) and the code went in.

So, now, a few weeks after that, I got around to dealing with that bug (because hey, this was going to be an easy fix that’ll give me a nice sense of accomplishment). A quick look in the Field_enum code raised my suspicions of bugs… I initially wondered if we’d get any error message if a StorageEngine returned a table definition that had too many ENUM elements (for example, 70,000). So, I added a table to the tableprototester plugin (a simple dummy engine that is loaded for testing the parsing of specially constructed table messages) that had 70,000 elements for a single ENUM column. It didn’t throw an error. Darn. It did, however, have an incredibly large result for SHOW CREATE TABLE.

Often with bugs like this I may try to see if the problem is something inherited from MySQL. I’ll often file a bug with MySQL as well if that’s the case. If I can, I’ll sometimes attach the associated patch from Drizzle that fixes the bug, sometimes with a patch directly for and tested on MySQL (if it’s not going to take me too long). If these patches are ever applied is a whole other thing – and sometimes you get things like “each engine is meant to have auto_increment behave differently!” – which doesn’t inspire confidence.

But anyway, the MySQL limit is somewhere between 10850 and 10900. This is not at all what’s documented. I’ve filed the appropriate bug (Bug #54194) with reproducible test case and the bit of problematic code. It turns out that this is (yet another) limit of the FRM file. The limit is “about 64k FRM”. The bit of code in MySQL that was doing the checking for the ENUM limit was this:



/* Hack to avoid bugs with small static rows in MySQL */
  reclength=max(file->min_record_length(table_options),reclength);
  if (info_length+(ulong) create_fields.elements*FCOMP+288+
      n_length+int_length+com_length > 65535L || int_count > 255)
  {
    my_message(ER_TOO_MANY_FIELDS, ER(ER_TOO_MANY_FIELDS), MYF(0));
    DBUG_RETURN(1);
  }


So it’s no surprise to anyone how this specific limit (the number of elements in an ENUM) got missed when I converted Drizzle from using an FRM over to a protobuf based structure.


So a bunch of other cleanup later, a whole lot of extra testing and I can pretty confidently state that the ENUM type in Drizzle does work exactly how you think it would.


Either way, if you’re getting anywhere near 10,000 choices for an ENUM column you have no doubt already lost.










PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Stewart Smith";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:13;a:6:{s:4:"data";s:88:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:39:"ODTUG Kaleidoscope 2010: Best Practices";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:36:"http://www.pythian.com/news/?p=13859";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:73:"http://www.pythian.com/news/13859/odtug-kaleidoscope-2010-best-practices/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:2824:"Updated: 29-Jun-2010, 30-Jun-2010.
For me, ODTUG Kaleidoscope 2010 started on Friday with the ACE Directors briefing. Best practices topic was touched there slightly and I twitted about it. I decided that the feedback deserves a blog post so I’m simply quoting the conversation here. If you have anything to add, you know where to find the comment box.


alexgorbachev:
best practices should be forbidden or rather renamed to blue-prints #ACED
Fri Jun 25 16:26:26 +0000 2010




GregRahn: @alexgorbachev IMHO “Best Practices” are often sought after as an alternative to thinking and understanding.  Two key components there!
Fri Jun 25 16:33:07 +0000 2010




CaryMillsap: Amen. RT @alexgorbachev: best practices should be forbidden or rather renamed to blue-prints #ACED
Fri Jun 25 16:33:43 +0000 2010



simon_haslam: Not sure but the term is overused RT @alexgorbachev best practices should be forbidden or rather renamed to blue-prints #ACED
Fri Jun 25 18:05:42 +0000 2010



debralilley: @alexgorbachev  #bestpractice  is WIP – easier to collect and publish initially but much harder to keep up but we should strive for it #ACED
Fri Jun 25 18:30:18 +0000 2010



CaryMillsap: @debralilley @myfear Designing “practices” is like designing library functions: generalizing means parameterizing. #aced
Fri Jun 25 19:16:49 +0000 2010



dannorris: @alexgorbachev re: best practices; what's in a name? Call them “stuff to do that avoids bad things” Note: I am not in marketing :)
Fri Jun 25 19:34:40 +0000 2010



CaryMillsap: @dannorris It's much easier to document the generally bad ideas than to document the generally good ones. #TenCommandments  #aced
Fri Jun 25 19:37:52 +0000 2010

CaryMillsap: @dannorris s/easier/more practical/g
Fri Jun 25 20:16:02 +0000 2010



sheeri: “Best practices” should be thought of as “up for renewal” often.  Often, “best practice + time = myth”. #ACED
Fri Jun 25 20:43:08 +0000 2010



oraclebase: To paraphrase Tom Kyte, the list of caveats is always longer than the best practice itself. #ACED
Fri Jun 25 20:49:15 +0000 2010



hyounpark_AG: Very true. Best != Permanent or immutable RT: @sheeri: “Best practices” should be thought of as “up for renewal” often. #ACED
Fri Jun 25 20:50:55 +0000 2010



CaryMillsap: “Best practice” the way to say “good idea” (or “MY idea”) when you want to stop people from thinking critically.
Sat Jun 26 14:51:14 +0000 2010


 coins term “Minimally Sufficient Practice” (MSP) to combat #bestpractice  epidemic. Yes.

CaryMillsap: On Oracle-L, @pudge1954 coins term “Minimally Sufficient Practice” (MSP) to combat #bestpractice  epidemic. Yes.
Sat Jun 26 14:51:14 +0000 2010


";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 03:02:12 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:10:{i:0;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:17:"Non-Tech Articles";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:6:"Oracle";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:10:"SQL Server";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:14:"Technical Blog";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:14:"best practices";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:11:"conferences";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:12:"kaleidoscope";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:5:"odtug";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:9;a:5:{s:4:"data";s:7:"twitter";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:10422:"
Updated: 29-Jun-2010, 30-Jun-2010.


For me, ODTUG Kaleidoscope 2010 started on Friday with the ACE Directors briefing. Best practices topic was touched there slightly and I twitted about it. I decided that the feedback deserves a blog post so I’m simply quoting the conversation here. If you have anything to add, you know where to find the comment box.






alexgorbachev:

best practices should be forbidden or rather renamed to blue-prints #ACED

Fri Jun 25 16:26:26 +0000 2010









GregRahn: @alexgorbachev IMHO “Best Practices” are often sought after as an alternative to thinking and understanding.  Two key components there!

Fri Jun 25 16:33:07 +0000 2010










CaryMillsap: Amen. RT @alexgorbachev: best practices should be forbidden or rather renamed to blue-prints #ACED

Fri Jun 25 16:33:43 +0000 2010








simon_haslam: Not sure but the term is overused RT @alexgorbachev best practices should be forbidden or rather renamed to blue-prints #ACED

Fri Jun 25 18:05:42 +0000 2010








debralilley: @alexgorbachev  #bestpractice  is WIP – easier to collect and publish initially but much harder to keep up but we should strive for it #ACED

Fri Jun 25 18:30:18 +0000 2010








CaryMillsap: @debralilley @myfear Designing “practices” is like designing library functions: generalizing means parameterizing. #aced

Fri Jun 25 19:16:49 +0000 2010








dannorris: @alexgorbachev re: best practices; what's in a name? Call them “stuff to do that avoids bad things” Note: I am not in marketing :)

Fri Jun 25 19:34:40 +0000 2010








CaryMillsap: @dannorris It's much easier to document the generally bad ideas than to document the generally good ones. #TenCommandments  #aced

Fri Jun 25 19:37:52 +0000 2010




CaryMillsap: @dannorris s/easier/more practical/g

Fri Jun 25 20:16:02 +0000 2010








sheeri: “Best practices” should be thought of as “up for renewal” often.  Often, “best practice + time = myth”. #ACED

Fri Jun 25 20:43:08 +0000 2010








oraclebase: To paraphrase Tom Kyte, the list of caveats is always longer than the best practice itself. #ACED

Fri Jun 25 20:49:15 +0000 2010








hyounpark_AG: Very true. Best != Permanent or immutable RT: @sheeri: “Best practices” should be thought of as “up for renewal” often. #ACED

Fri Jun 25 20:50:55 +0000 2010








CaryMillsap: “Best practice” the way to say “good idea” (or “MY idea”) when you want to stop people from thinking critically.

Sat Jun 26 14:51:14 +0000 2010






 coins term “Minimally Sufficient Practice” (MSP) to combat #bestpractice  epidemic. Yes.




CaryMillsap: On Oracle-L, @pudge1954 coins term “Minimally Sufficient Practice” (MSP) to combat #bestpractice  epidemic. Yes.

Sat Jun 26 14:51:14 +0000 2010







PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:14:"Alex Gorbachev";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:14;a:6:{s:4:"data";s:63:"
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:32:"6to4: Easing the IPv6 transition";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:33:"http://openquery.com/blog/?p=1289";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:53:"http://openquery.com/blog/6to4-easing-ipv6-transition";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:9567:"With the exhaustion of IPv4 address space looming sometime in 2012; probably earlier rather than later, it makes sense to ease on into IPv6 land.  Without straying into tunnel broking and endpoint shenanigans 6to4 is a method of wrapping up IPv6 inside of IPv4.
(note that MySQL does not currently support IPv6 itself – but what we’re discussing here is about externally facing systems, like your web/application servers)
6to4 performs three functions:

Allocates an IPv6 address block to any host/network that has a global IPv4 address.
Wraps up IPv6 packets inside IPv4 packets for transmission over IPv4 using 6in4 (traffic is sent over IPv4 inside IPv4 packets whose IP headers have the IP protocol number  set to 41; IPv6-in-IPv4. ) 6to4 makes use of IP protocol 41 too, but instead of static endpoints, the endpoint IPv4 address is sourced from IPv6 addresses within the IPv6 packet header.
Routes traffic between 6to4 and “native” IPv6 networks.

As such its pretty easy to implement, especially on our good friend Debian (and its better looking cousin Ubuntu).
I am going to step through setting up a Debian  host at Linode.
Step 1 Check your Kernel
Now, the first caveat is that you must be running a 2.6.20+ kernel (At the time of writing the latest linode kernel for Debian was : 2.6 Paravirt (2.6.34-x86_64-linode)). The default ‘Etch’ release kernel (2.6.18) supports IPv6 but woefully implements IPv6 stateful connection tracking, which is just not good enough for a decent firewall. If you have a look under your Linode Configuration Profile you can see what Kernel you are running, and change it to one that is supported; obviously a reboot would be in order if you change it. The linode kernels have IPV6 support compiled in.
Step 2 Calculate your new IPv6 address
Any IPv6 address that begins with the 2002::/16 prefix is known as a 6to4 address, as opposed to a native IPv6 address which does not use that prefix. The Internet Assigned Numbers Authority (IANA: www.iana.org) has set aside this address space just for 6to4. IPv6 addresses are assigned based upon your IPv4 address; for instance, 74.207.254.16 would become 2002:4acf:fe10::/48
We need some tools to help us calculate our IPV6 address, luckily there is a package for this
$ sudo apt-get ipv6calc
Now its a matter of plugging in your IPv4 address into ipv6calc to determine your reserved IPv6 address range.
$ ipv6calc -q --action conv6to4 --in ipv4 74.207.254.16 --out ipv6
and voila your IPv6 address range appears:
2002:4acf:fe10:: (/48)
You get given an address range with a prefix length of 48 bits, which leaves room for a 16-bit subnet field and a 64 bit host address within the subnet.
Step 3 Update your interface configuration
You now need to edit your network configuration file /etc/network/interfaces file
auto tun6to4
iface tun6to4 inet6 v4tunnel
address 2002:4acf:fe10::1
netmask 16
gateway ::192.88.99.1
endpoint any
local 74.207.254.16 #fits address
auto tun6to4 # make sure this interface comes up on boot
 iface tun6to4 inet6 v4tunnel
 address 2002:4acf:fe10::1 #first host in this address range
 netmask 16
 gateway ::192.88.99.1 #special anycast address for 6to4 (2002:c058:6301::)
 endpoint any
 local 74.207.254.16
 mtu 1472
 ttl 255
Restart your interfaces:
$sudo /etc/init.d/networking restart
Step 4 Update IPv6 Firewall script/rules
Now it’s fairly important (read as critical) to firewall IPv6 stuff as it is with IPv4. Here is a small sample of a firewall that will at the very least not leave you hanging in the breeze. Needless to say you can add your own rules and make this as complex as you need.
# Initialize all the chains by removing all the rules
iptables --flush
iptables -t nat --flush
iptables -t mangle --flush
ip6tables --flush
ip6tables -t mangle --flush
# The loopback interface should accept all traffic
iptables -A INPUT  -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A OUTPUT -o lo -j ACCEPT
#Allow IPV6 packets to come over the tunnel
iptables -A INPUT -p ipv6 -i eth0 -j ACCEPT
iptables -A OUTPUT -p ipv6 -o eth0 -j ACCEPT
# Allow outbound DNS queries from the FW and the replies too
iptables -A OUTPUT -p udp -o eth0 --dport 53 --sport 1024:65535 -j ACCEPT
iptables -A INPUT -p udp -i eth0 --sport 53 --dport 1024:65535  -j ACCEPT
ip6tables -A OUTPUT -p udp -o tun6to4 --dport 53 --sport 1024:65535 -j ACCEPT
ip6tables -A INPUT -p udp -i tun6to4 --sport 53 --dport 1024:65535  -j ACCEPT
# Accept and reply to ICMP ping
iptables -A OUTPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-reply -j ACCEPT
# IMPORTANT!!!! Allow all icmpv6 because they make IPV6 work
ip6tables -A OUTPUT -p icmpv6 -j ACCEPT
ip6tables -A INPUT -p icmpv6 -j ACCEPT
# Allow previously established connections
iptables -A OUTPUT -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
ip6tables -A OUTPUT -o tun6to4 -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow port 80 (www) and 51515 (SSH) connections to the firewall
iptables -A INPUT -p tcp -i eth0 --dport 51515 --sport 1024:65535 -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -i eth0 --dport 443 --sport 1024:65535 -m state --state NEW -j ACCEPT
iptables -A INPUT -p tcp -i eth0 --dport 80 --sport 1024:65535 -m state --state NEW -j ACCEPT
ip6tables -A INPUT -p tcp -i tun6to4 --dport 51515 --sport 1024:65535 -m state --state NEW -j ACCEPT
ip6tables -A INPUT -p tcp -i tun6to4 --dport 443 --sport 1024:65535 -m state --state NEW -j ACCEPT
ip6tables -A INPUT -p tcp -i tun6to4 --dport 80 --sport 1024:65535 -m state --state NEW -j ACCEPT
# Allow port 80 (www) and 443 (https) connections from the firewall
iptables -A OUTPUT -j ACCEPT -m state --state NEW,ESTABLISHED,RELATED -o eth0 -p tcp -m multiport --dport 51515,80,443 -m multiport --sport 1024:65535
ip6tables -A OUTPUT -j ACCEPT -m state --state NEW,ESTABLISHED,RELATED -o tun6to4 -p tcp -m multiport --dport 51515,80,443 -m multiport  --sport 1024:65535
# Allow previously established connections
iptables -A INPUT -j ACCEPT -m state --state ESTABLISHED,RELATED -i eth0 -p tcp
ip6tables -A INPUT -j ACCEPT -m state --state ESTABLISHED,RELATED -i tun6to4 -p tcp
# The policy should be to drop it
iptables -A INPUT -j DROP
iptables -A OUTPUT -j DROP
iptables -A FORWARD -j DROP
ip6tables -A INPUT -j DROP
ip6tables -A OUTPUT -j DROP
ip6tables -A FORWARD -j DROP
I usually create a directory called /etc/iptables  (owner root:root  / permissions 750) and drop  firewall up and down scripts in there.
Then it is  a simple matter of adding the following scripts to the bottom of your eth0 interface definition stanza in /etc/network/interfaces to invoke them on boot or whenever:
pre-up /etc/iptables/firewall_up.sh
post-down /etc/iptables/firewall_down.sh
pre-up /etc/iptables/firewall_up.sh
post-down /etc/iptables/firewall_down.sh
IMPORTANT: Just a quick note don’t block icmpv6 because it is the glue that holds IPv6 together.


Step 5 Setup Forward DNS
I am not going to over explain this one because everyone has an opinion on how to setup DNS but in essence you need to add a line like this to your zone file. There are plenty articles outlining this stuff.
 
hyosine			AAAA	2002:4acf:fe10::1
Step 6 Setup Reverse DNS
You now need to setup  reverse DNS for your address, so using our example of 2002:4acf:fe10 you will have to configure the zone of ”0.0.0.0.0.1.e.f.f.c.a.4.2.0.0.2.ip6.arpa” in your name servers.  The zone should have PTR records for your hosts just like an in-addr.arpa zone for IPv4, but with hex digits of the IPv6 address backwards, separated by dots. Using our example, the 6to4 host will have a ::1 suffix, so a reverse DNS record looks like:
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.e.f.f.c.a.4.2.0.0.2.ip6.arpa. PTR hyosine.openquery.com.
You will need to register this zone and its servers with the 6to4 reverse zone authority. eg https://6to4.nro.net/
Step 7  Test
The ping6 utility is probably best to test whether your host is now working. It’s probably best to try the IPv6 address first:

$ ping6 2002:4acf:fe10::1 
 PING 2002:4acf:fe10::1(2002:4acf:fe10::1) 56 data bytes 
 64 bytes from 2002:4acf:fe10::1: icmp_seq=1 ttl=60 time=1.59 ms 
 64 bytes from 2002:4acf:fe10::1: icmp_seq=2 ttl=60 time=1.42 ms 
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR 6to4.example.com.
With that record inside the above zone, the full record would be
Now you can try with the DNS name you just setup.
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.7.c.2.0.8.a.0.c.2.0.0.2.ip6.arpa. PTR 6to4.example.com.
$ ping6 hyosine.cloudcaster.com
PING hyosine.cloudcaster.com(2002:4acf:fe10::1) 56 data bytes
64 bytes from 2002:4acf:fe10::1: icmp_seq=1 ttl=60 time=1.41 ms
64 bytes from 2002:4acf:fe10::1: icmp_seq=2 ttl=60 time=1.34 ms
$ ping6 hyosine.cloudcaster.com
PING hyosine.cloudcaster.com(2002:4acf:fe10::1) 56 data bytes
64 bytes from 2002:4acf:fe10::1: icmp_seq=1 ttl=60 time=1.41 ms
64 bytes from 2002:4acf:fe10::1: icmp_seq=2 ttl=60 time=1.34 ms
Lastly, you need to register this zone and its servers with the 6to4 reverse zone authority. Note that when you visit that site, you’ll get an SSL certificate warning. This is normal. You need to visit this site using IPv6 from the actual 6to4 zone you’re trying to register. Follow the form to set up the nameservers for the zone and that’s it!
    ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Tue, 29 Jun 2010 00:48:11 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:5:{i:0;a:5:{s:4:"data";s:13:"Uncategorized";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:4:"ipv6";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:7:"mariadb";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:10:"Open Query";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:11838:"
With the exhaustion of IPv4 address space looming sometime in 2012; probably earlier rather than later, it makes sense to ease on into IPv6 land.  Without straying into tunnel broking and endpoint shenanigans 6to4 is a method of wrapping up IPv6 inside of IPv4.


(note that MySQL does not currently support IPv6 itself – but what we’re discussing here is about externally facing systems, like your web/application servers)


6to4 performs three functions:


    

  1. Allocates an IPv6 address block to any host/network that has a global IPv4 address.
    2. 

  2. Wraps up IPv6 packets inside IPv4 packets for transmission over IPv4 using 6in4 (traffic is sent over IPv4 inside IPv4 packets whose IP headers have the IP protocol number  set to 41; IPv6-in-IPv4. ) 6to4 makes use of IP protocol 41 too, but instead of static endpoints, the endpoint IPv4 address is sourced from IPv6 addresses within the IPv6 packet header.
    3. 

  3. Routes traffic between 6to4 and “native” IPv6 networks.
    4. 



As such its pretty easy to implement, especially on our good friend Debian (and its better looking cousin Ubuntu).


I am going to step through setting up a Debian  host at Linode.


Step 1 Check your Kernel


Now, the first caveat is that you must be running a 2.6.20+ kernel (At the time of writing the latest linode kernel for Debian was : 2.6 Paravirt (2.6.34-x86_64-linode)). The default ‘Etch’ release kernel (2.6.18) supports IPv6 but woefully implements IPv6 stateful connection tracking, which is just not good enough for a decent firewall. If you have a look under your Linode Configuration Profile you can see what Kernel you are running, and change it to one that is supported; obviously a reboot would be in order if you change it. The linode kernels have IPV6 support compiled in.


Step 2 Calculate your new IPv6 address


Any IPv6 address that begins with the 2002::/16 prefix is known as a 6to4 address, as opposed to a native IPv6 address which does not use that prefix. The Internet Assigned Numbers Authority (IANA: www.iana.org) has set aside this address space just for 6to4. IPv6 addresses are assigned based upon your IPv4 address; for instance, 74.207.254.16 would become 2002:4acf:fe10::/48


We need some tools to help us calculate our IPV6 address, luckily there is a package for this


$ sudo apt-get ipv6calc


Now its a matter of plugging in your IPv4 address into ipv6calc to determine your reserved IPv6 address range.


$ ipv6calc -q --action conv6to4 --in ipv4 74.207.254.16 --out ipv6


and voila your IPv6 address range appears:


2002:4acf:fe10:: (/48)


You get given an address range with a prefix length of 48 bits, which leaves room for a 16-bit subnet field and a 64 bit host address within the subnet.


Step 3 Update your interface configuration


You now need to edit your network configuration file /etc/network/interfaces file


auto tun6to4


iface tun6to4 inet6 v4tunnel


address 2002:4acf:fe10::1


netmask 16


gateway ::192.88.99.1


endpoint any


local 74.207.254.16 #fits address


auto tun6to4 # make sure this interface comes up on boot


 iface tun6to4 inet6 v4tunnel


 address 2002:4acf:fe10::1 #first host in this address range


 netmask 16


 gateway ::192.88.99.1 #special anycast address for 6to4 (2002:c058:6301::)


 endpoint any


 local 74.207.254.16


 mtu 1472


 ttl 255


Restart your interfaces:


$sudo /etc/init.d/networking restart


Step 4 Update IPv6 Firewall script/rules


Now it’s fairly important (read as critical) to firewall IPv6 stuff as it is with IPv4. Here is a small sample of a firewall that will at the very least not leave you hanging in the breeze. Needless to say you can add your own rules and make this as complex as you need.


# Initialize all the chains by removing all the rules


iptables --flush


iptables -t nat --flush


iptables -t mangle --flush


ip6tables --flush


ip6tables -t mangle --flush


# The loopback interface should accept all traffic


iptables -A INPUT  -i lo -j ACCEPT


iptables -A OUTPUT -o lo -j ACCEPT


ip6tables -A INPUT -i lo -j ACCEPT


ip6tables -A OUTPUT -o lo -j ACCEPT


#Allow IPV6 packets to come over the tunnel


iptables -A INPUT -p ipv6 -i eth0 -j ACCEPT


iptables -A OUTPUT -p ipv6 -o eth0 -j ACCEPT


# Allow outbound DNS queries from the FW and the replies too


iptables -A OUTPUT -p udp -o eth0 --dport 53 --sport 1024:65535 -j ACCEPT


iptables -A INPUT -p udp -i eth0 --sport 53 --dport 1024:65535  -j ACCEPT


ip6tables -A OUTPUT -p udp -o tun6to4 --dport 53 --sport 1024:65535 -j ACCEPT


ip6tables -A INPUT -p udp -i tun6to4 --sport 53 --dport 1024:65535  -j ACCEPT


# Accept and reply to ICMP ping


iptables -A OUTPUT -p icmp --icmp-type echo-request -j ACCEPT


iptables -A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT


iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT


iptables -A INPUT -p icmp --icmp-type echo-reply -j ACCEPT


# IMPORTANT!!!! Allow all icmpv6 because they make IPV6 work


ip6tables -A OUTPUT -p icmpv6 -j ACCEPT


ip6tables -A INPUT -p icmpv6 -j ACCEPT


# Allow previously established connections


iptables -A OUTPUT -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT


ip6tables -A OUTPUT -o tun6to4 -m state --state ESTABLISHED,RELATED -j ACCEPT


# Allow port 80 (www) and 51515 (SSH) connections to the firewall


iptables -A INPUT -p tcp -i eth0 --dport 51515 --sport 1024:65535 -m state --state NEW -j ACCEPT


iptables -A INPUT -p tcp -i eth0 --dport 443 --sport 1024:65535 -m state --state NEW -j ACCEPT


iptables -A INPUT -p tcp -i eth0 --dport 80 --sport 1024:65535 -m state --state NEW -j ACCEPT


ip6tables -A INPUT -p tcp -i tun6to4 --dport 51515 --sport 1024:65535 -m state --state NEW -j ACCEPT


ip6tables -A INPUT -p tcp -i tun6to4 --dport 443 --sport 1024:65535 -m state --state NEW -j ACCEPT


ip6tables -A INPUT -p tcp -i tun6to4 --dport 80 --sport 1024:65535 -m state --state NEW -j ACCEPT


# Allow port 80 (www) and 443 (https) connections from the firewall


iptables -A OUTPUT -j ACCEPT -m state --state NEW,ESTABLISHED,RELATED -o eth0 -p tcp -m multiport --dport 51515,80,443 -m multiport --sport 1024:65535


ip6tables -A OUTPUT -j ACCEPT -m state --state NEW,ESTABLISHED,RELATED -o tun6to4 -p tcp -m multiport --dport 51515,80,443 -m multiport  --sport 1024:65535


# Allow previously established connections


iptables -A INPUT -j ACCEPT -m state --state ESTABLISHED,RELATED -i eth0 -p tcp


ip6tables -A INPUT -j ACCEPT -m state --state ESTABLISHED,RELATED -i tun6to4 -p tcp


# The policy should be to drop it


iptables -A INPUT -j DROP


iptables -A OUTPUT -j DROP


iptables -A FORWARD -j DROP


ip6tables -A INPUT -j DROP


ip6tables -A OUTPUT -j DROP


ip6tables -A FORWARD -j DROP


I usually create a directory called /etc/iptables  (owner root:root  / permissions 750) and drop  firewall up and down scripts in there.


Then it is  a simple matter of adding the following scripts to the bottom of your eth0 interface definition stanza in /etc/network/interfaces to invoke them on boot or whenever:


pre-up /etc/iptables/firewall_up.sh


post-down /etc/iptables/firewall_down.sh


pre-up /etc/iptables/firewall_up.sh


post-down /etc/iptables/firewall_down.sh


IMPORTANT: Just a quick note don’t block icmpv6 because it is the glue that holds IPv6 together.







Step 5 Setup Forward DNS


I am not going to over explain this one because everyone has an opinion on how to setup DNS but in essence you need to add a line like this to your zone file. There are plenty articles outlining this stuff.


 


hyosine			AAAA	2002:4acf:fe10::1


Step 6 Setup Reverse DNS


You now need to setup  reverse DNS for your address, so using our example of 2002:4acf:fe10 you will have to configure the zone of ”0.0.0.0.0.1.e.f.f.c.a.4.2.0.0.2.ip6.arpa” in your name servers.  The zone should have PTR records for your hosts just like an in-addr.arpa zone for IPv4, but with hex digits of the IPv6 address backwards, separated by dots. Using our example, the 6to4 host will have a ::1 suffix, so a reverse DNS record looks like:


1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.e.f.f.c.a.4.2.0.0.2.ip6.arpa. PTR hyosine.openquery.com.


You will need to register this zone and its servers with the 6to4 reverse zone authority. eg https://6to4.nro.net/


Step 7  Test


The ping6 utility is probably best to test whether your host is now working. It’s probably best to try the IPv6 address first:



$ ping6 2002:4acf:fe10::1 


 PING 2002:4acf:fe10::1(2002:4acf:fe10::1) 56 data bytes 


 64 bytes from 2002:4acf:fe10::1: icmp_seq=1 ttl=60 time=1.59 ms 


 64 bytes from 2002:4acf:fe10::1: icmp_seq=2 ttl=60 time=1.42 ms 


1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 PTR 6to4.example.com.


With that record inside the above zone, the full record would be


Now you can try with the DNS name you just setup.


1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.7.c.2.0.8.a.0.c.2.0.0.2.ip6.arpa. PTR 6to4.example.com.


$ ping6 hyosine.cloudcaster.com


PING hyosine.cloudcaster.com(2002:4acf:fe10::1) 56 data bytes


64 bytes from 2002:4acf:fe10::1: icmp_seq=1 ttl=60 time=1.41 ms


64 bytes from 2002:4acf:fe10::1: icmp_seq=2 ttl=60 time=1.34 ms


$ ping6 hyosine.cloudcaster.com


PING hyosine.cloudcaster.com(2002:4acf:fe10::1) 56 data bytes


64 bytes from 2002:4acf:fe10::1: icmp_seq=1 ttl=60 time=1.41 ms


64 bytes from 2002:4acf:fe10::1: icmp_seq=2 ttl=60 time=1.34 ms


Lastly, you need to register this zone and its servers with the 6to4 reverse zone authority. Note that when you visit that site, you’ll get an SSL certificate warning. This is normal. You need to visit this site using IPv6 from the actual 6to4 zone you’re trying to register. Follow the form to set up the nameservers for the zone and that’s it!


    

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:10:"Open Query";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:15;a:6:{s:4:"data";s:58:"
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:35:"Maria Engine Performance once again";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:70:"tag:blogger.com,1999:blog-9144505959002328789.post-8780596335401843939";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:84:"http://karlssonondatabases.blogspot.com/2010/06/mariadb-performancxe-once-again.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:1674:"Note: Article has been edited as I was confusing MarisDB and the Maria Stoare Engine!Just for the heck of it, I decided to try a very simple benchmark on the Maria Storage Engine again. This time, I'm using a simple SELECT. The table I use looks like this:CREATE TABLE `t1` (  `c1` int(11) NOT NULL AUTO_INCREMENT,  `c2` char(100) DEFAULT NULL,  PRIMARY KEY (`c1`));Which should be simple enough. I fill this with 1.3 millions rows, with c1 having consequitive numbers from 1 and up. My benchmark consists of random SELECTs of the c2 column from this table, using a primary key lookup on c1. I run this on 400 concurrent threads with each thread doing 1000 SELECTs. Ignoring the actual values, MyISAM and InnoDB come out pretty close in performance, with InnoDB slightly behind. Which is reasonable. Maria is at less than half the performance of MyISAM though. This is worrying. And I know what you say, Maria is new, performance comes later. I say performance comes mainly from design. As Maria does not have proper transactions yet, only row level transactions (not statement level. Try to stop a INSERT INTO ... SELECT FROM and you see what I mean).I was hoping for more from Maria. As I tested MariaDB 5.1.47, I could also test PBXT. This particular test is hardly targeted at what PBXT is good at though, and I didn't even have the energy to wait for it to finish a single round of this test.In this case, in difference to before, I did some basic tuning. Maria proved difficult here, as there is no documentation on the parameters, at least none that I could find.Note: All tests run using MariaDB 5.1.47, including the ones involving InnoDB, MyISAM and PBXT./Karlsson";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 21:10:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:11:"performance";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:6:"innodb";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:7:"mariadb";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:2234:"Note: Article has been edited as I was confusing MarisDB and the Maria Stoare Engine!

Just for the heck of it, I decided to try a very simple benchmark on the Maria Storage Engine again. This time, I'm using a simple SELECT. The table I use looks like this:
CREATE TABLE `t1` (
  `c1` int(11) NOT NULL AUTO_INCREMENT,
  `c2` char(100) DEFAULT NULL,
  PRIMARY KEY (`c1`)
);
Which should be simple enough. I fill this with 1.3 millions rows, with c1 having consequitive numbers from 1 and up. My benchmark consists of random SELECTs of the c2 column from this table, using a primary key lookup on c1. I run this on 400 concurrent threads with each thread doing 1000 SELECTs. Ignoring the actual values, MyISAM and InnoDB come out pretty close in performance, with InnoDB slightly behind. Which is reasonable. Maria is at less than half the performance of MyISAM though. This is worrying. And I know what you say, Maria is new, performance comes later. I say performance comes mainly from design. As Maria does not have proper transactions yet, only row level transactions (not statement level. Try to stop a INSERT INTO ... SELECT FROM and you see what I mean).

I was hoping for more from Maria. As I tested MariaDB 5.1.47, I could also test PBXT. This particular test is hardly targeted at what PBXT is good at though, and I didn't even have the energy to wait for it to finish a single round of this test.

In this case, in difference to before, I did some basic tuning. Maria proved difficult here, as there is no documentation on the parameters, at least none that I could find.

Note: All tests run using MariaDB 5.1.47, including the ones involving InnoDB, MyISAM and PBXT.

/Karlsson

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:15:"Anders Karlsson";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:16;a:6:{s:4:"data";s:58:"
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:47:"MySQL Sandbox embraces Python and meets Cluster";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:59:"tag:blogger.com,1999:blog-16959946.post-8101332860596809352";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:84:"http://datacharmer.blogspot.com/2010/06/mysql-sandbox-embraces-python-and-meets.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:6763:" If you have tried Quick start guides: MySQL cluster in 10 minutes, you may have realized that it is really quick and easy.However, it leaves some typing to be done.Users of MySQL Sandbox have a horror of repetitive typing, and this got me thinking. "Could I integrate MySQL Sandbox and Cluster?"The answer was: "Sure."But then I started thinking of all the minor and major changes that I wanted to do to the Sandbox and have delayed for too long. What I need, is a radical refactoring. And then I remembered that it has been almost two years since I learned a new programming language and that perhaps I could expand my horizons and the Sandbox architecture at once. Thus, thanks to an irresistible offer from O'reilly about ebooks, last week I bought both Learning Python, fourth edition and Programming Python, Third edition. During the week end I produced my first tool: a Python script that installs and starts a small cluster, following the instructions given in the MySQL Cluster quick start guides. The script unpacks the cluster tarball, installs a server sandbox from it, then starts the cluster nodes and the MySQL server, and then it monitors the cluster until all the nodes are connected before finally testing the server.Here is a sample run:$ make_cluster.py ~/Downloads/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64.tar.gz++ tar -xzf ~/Downloads/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64.tar.gz++ low_level_make_sandbox --basedir=$HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64 --sandbox_directory=mcluster --install_version=5.1 --sandbox_port=5144 --no_ver_after_name --no_run --force --my_clause=log-error=msandbox.err --my_clause=ndbcluster     The MySQL Sandbox,  version 3.0.12    (C) 2006-2010 Giuseppe Maxiainstalling with the following parameters:upper_directory                = $HOME/sandboxessandbox_directory              = mclustersandbox_port                   = 5144check_port                     = no_check_port                  = datadir_from                   = scriptinstall_version                = 5.1basedir                        = $HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64tmpdir                         = my_file                        = operating_system_user          = gmaxdb_user                        = msandboxdb_password                    = msandboxmy_clause                      = log-error=msandbox.err ; ndbclusterprompt_prefix                  = mysqlprompt_body                    =  [\h] {\u} (\d) > force                          = 1no_ver_after_name              = 1verbose                        = load_grants                    = 1no_load_grants                 = no_run                         = 1no_show                        = loading grants.. sandbox server startedstopping serverYour sandbox server was installed in $HOME/sandboxes/mcluster++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgmd -f $HOME/sandboxes/mcluster/my_cluster/conf/config.ini --initial --configdir=$HOME/sandboxes/mcluster/my_cluster/conf/2010-06-28 21:29:57 [MgmtSrvr] INFO     -- NDB Cluster Management Server. mysql-5.1.44 ndb-7.1.4b2010-06-28 21:29:57 [MgmtSrvr] INFO     -- Reading cluster configuration from '$HOME/sandboxes/mcluster/my_cluster/conf/config.ini'++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndbd -c localhost:11862010-06-28 21:29:57 [ndbd] INFO     -- Configuration fetched from 'localhost:1186', generation: 1++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndbd -c localhost:11862010-06-28 21:29:57 [ndbd] INFO     -- Configuration fetched from 'localhost:1186', generation: 1++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgm -e showConnected to Management Server at: localhost:1186Cluster Configuration---------------------[ndbd(NDB)] 2 node(s)id=3 (not connected, accepting connect from localhost)id=4 (not connected, accepting connect from localhost)[ndb_mgmd(MGM)] 1 node(s)id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)[mysqld(API)] 1 node(s)id=50 (not connected, accepting connect from any host)++ $HOME/sandboxes/mcluster/clear++ $HOME/sandboxes/mcluster/start... sandbox server startedPlease wait. Giving the cluster time to catch upConnected to Management Server at: localhost:1186Cluster Configuration---------------------[ndbd(NDB)] 2 node(s)id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)[ndb_mgmd(MGM)] 1 node(s)id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)[mysqld(API)] 1 node(s)id=50 (not connected, accepting connect from any host)It may take up to 2 minutes to initialize ... ( 0 )Connected to Management Server at: localhost:1186Cluster Configuration---------------------[ndbd(NDB)] 2 node(s)id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)[ndb_mgmd(MGM)] 1 node(s)id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)[mysqld(API)] 1 node(s)id=50 (not connected, accepting connect from any host)It may take up to 2 minutes to initialize ... ( 5 )Connected to Management Server at: localhost:1186Cluster Configuration---------------------[ndbd(NDB)] 2 node(s)id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)[ndb_mgmd(MGM)] 1 node(s)id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)[mysqld(API)] 1 node(s)id=50 (not connected, accepting connect from any host)[...]It may take up to 2 minutes to initialize ... ( 100 )Connected to Management Server at: localhost:1186Cluster Configuration---------------------[ndbd(NDB)] 2 node(s)id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, Nodegroup: 0, Master)id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, Nodegroup: 0)[ndb_mgmd(MGM)] 1 node(s)id=1 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4)[mysqld(API)] 1 node(s)id=50 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4)++ $HOME/sandboxes/mcluster/use -vvv -e "create table test.t1(i int not null primary key)engine=ndb"--------------create table test.t1(i int not null primary key)engine=ndb--------------Query OK, 0 rows affected (0.45 sec)++ $HOME/sandboxes/mcluster/use -vvv -e "show create table test.t1\G"--------------show create table test.t1--------------*************************** 1. row ***************************       Table: t1Create Table: CREATE TABLE `t1` (  `i` int(11) NOT NULL,  PRIMARY KEY (`i`)) ENGINE=ndbcluster DEFAULT CHARSET=latin11 row in set (0.00 sec)to shut down the cluster, type:$HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgm -e shutdownIt works! Ans this is a good start to make me feel confident with Python, which I will use to develop MySQL Sandbox version 4. This cluster snippet will probably be made from scratch once the new architecture is in place. For now, it was enough to get the feeling of the language.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 20:34:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:7:"sandbox";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:7:"cluster";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:6:"python";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:8790:"
 If you have tried Quick start guides: MySQL cluster in 10 minutes, you may have realized that it is really quick and easy.
However, it leaves some typing to be done.
Users of MySQL Sandbox have a horror of repetitive typing, and this got me thinking. "Could I integrate MySQL Sandbox and Cluster?"
The answer was: "Sure."
But then I started thinking of all the minor and major changes that I wanted to do to the Sandbox and have delayed for too long. What I need, is a radical refactoring. 
And then I remembered that it has been almost two years since I learned a new programming language and that perhaps I could expand my horizons and the Sandbox architecture at once. 
Thus, thanks to an irresistible offer from O'reilly about ebooks, last week I bought both Learning Python, fourth edition and Programming Python, Third edition. 
During the week end I produced my first tool: a Python script that installs and starts a small cluster, following the instructions given in the MySQL Cluster quick start guides. The script unpacks the cluster tarball, installs a server sandbox from it, then starts the cluster nodes and the MySQL server, and then it monitors the cluster until all the nodes are connected before finally testing the server.
Here is a sample run:

$ make_cluster.py ~/Downloads/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64.tar.gz

++ tar -xzf ~/Downloads/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64.tar.gz

++ low_level_make_sandbox --basedir=$HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64 --sandbox_directory=mcluster --install_version=5.1 --sandbox_port=5144 --no_ver_after_name --no_run --force --my_clause=log-error=msandbox.err --my_clause=ndbcluster 
    The MySQL Sandbox,  version 3.0.12
    (C) 2006-2010 Giuseppe Maxia
installing with the following parameters:
upper_directory                = $HOME/sandboxes
sandbox_directory              = mcluster
sandbox_port                   = 5144
check_port                     = 
no_check_port                  = 
datadir_from                   = script
install_version                = 5.1
basedir                        = $HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64
tmpdir                         = 
my_file                        = 
operating_system_user          = gmax
db_user                        = msandbox
db_password                    = msandbox
my_clause                      = log-error=msandbox.err ; ndbcluster
prompt_prefix                  = mysql
prompt_body                    =  [\h] {\u} (\d) > 
force                          = 1
no_ver_after_name              = 1
verbose                        = 
load_grants                    = 1
no_load_grants                 = 
no_run                         = 1
no_show                        = 
loading grants
.. sandbox server started
stopping server
Your sandbox server was installed in $HOME/sandboxes/mcluster

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgmd -f $HOME/sandboxes/mcluster/my_cluster/conf/config.ini --initial --configdir=$HOME/sandboxes/mcluster/my_cluster/conf/
2010-06-28 21:29:57 [MgmtSrvr] INFO     -- NDB Cluster Management Server. mysql-5.1.44 ndb-7.1.4b
2010-06-28 21:29:57 [MgmtSrvr] INFO     -- Reading cluster configuration from '$HOME/sandboxes/mcluster/my_cluster/conf/config.ini'

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndbd -c localhost:1186
2010-06-28 21:29:57 [ndbd] INFO     -- Configuration fetched from 'localhost:1186', generation: 1

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndbd -c localhost:1186
2010-06-28 21:29:57 [ndbd] INFO     -- Configuration fetched from 'localhost:1186', generation: 1

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 (not connected, accepting connect from localhost)
id=4 (not connected, accepting connect from localhost)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)


++ $HOME/sandboxes/mcluster/clear

++ $HOME/sandboxes/mcluster/start
... sandbox server started
Please wait. Giving the cluster time to catch up
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)


It may take up to 2 minutes to initialize ... ( 0 )
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)


It may take up to 2 minutes to initialize ... ( 5 )
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)

[...]

It may take up to 2 minutes to initialize ... ( 100 )
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4)


++ $HOME/sandboxes/mcluster/use -vvv -e "create table test.t1(i int not null primary key)engine=ndb"
--------------
create table test.t1(i int not null primary key)engine=ndb
--------------
Query OK, 0 rows affected (0.45 sec)

++ $HOME/sandboxes/mcluster/use -vvv -e "show create table test.t1\G"
--------------
show create table test.t1
--------------

*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `i` int(11) NOT NULL,
  PRIMARY KEY (`i`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

to shut down the cluster, type:
$HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgm -e shutdown
It works! Ans this is a good start to make me feel confident with Python, which I will use to develop MySQL Sandbox version 4. This cluster snippet will probably be made from scratch once the new architecture is in place. For now, it was enough to get the feeling of the language.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:14:"Giuseppe Maxia";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:17;a:6:{s:4:"data";s:88:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:30:"MySQL Idiosyncrasies That Bite";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:38:"http://ronaldbradford.com/blog/?p=2991";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:73:"http://ronaldbradford.com/blog/mysql-idiosyncrasies-that-bite-2010-06-28/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:454:"The following are my slides that I presented at ODTUG Kaleidoscope 2010.  This presentation talks about the MySQL defaults including a non-transactional state, silent data truncations, date management, and transaction isolation options. These are all critical for data integrity and consistency.  I also cover in-depth topics including SQL_MODE, character sets and collations.
MySQL Idiosyncrasies That Bite
View more presentations from Ronald Bradford.
";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 17:38:49 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:10:{i:0;a:5:{s:4:"data";s:9:"Databases";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:17:"Kaleidoscope 2010";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:24:"Oracle/MySQL Conferences";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:12:"Professional";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:14:"data integrity";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:12:"Kaleidoscope";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:5:"odtug";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:6:"syntax";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:9;a:5:{s:4:"data";s:12:"transactions";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:992:"
The following are my slides that I presented at ODTUG Kaleidoscope 2010.  This presentation talks about the MySQL defaults including a non-transactional state, silent data truncations, date management, and transaction isolation options. These are all critical for data integrity and consistency.  I also cover in-depth topics including SQL_MODE, character sets and collations.


MySQL Idiosyncrasies That Bite

View more presentations from Ronald Bradford.



PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:15:"Ronald Bradford";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:18;a:6:{s:4:"data";s:38:"
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:45:"Supreme Court sidesteps software patent issue";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:51:"http://news.cnet.com/8301-30685_3-20009019-264.html";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:91:"http://news.cnet.com/8301-30685_3-20009019-264.html?part=rss&tag=feed&subj=DeepTech";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:138:"The court rejects one particular patent but doesn't provide new guidance about what types of business methods and software are patentable.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 16:43:32 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:351:"The court rejects one particular patent but doesn't provide new guidance about what types of business methods and software are patentable.
PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:9:"Matt Asay";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:19;a:6:{s:4:"data";s:43:"
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:35:"Lock-free vs. wait-free concurrency";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:36:"http://www.rethinkdb.com/blog/?p=409";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:73:"http://www.rethinkdb.com/blog/2010/06/lock-free-vs-wait-free-concurrency/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:2911:"There are two types of non-blocking thread synchronization algorithms – lock-free, and wait-free. Their meaning is often confused. In lock-free systems, while any particular computation may be blocked for some period of time, all CPUs are able to continue performing other computations. To put it differently, while a given thread might be blocked by other threads in a lock-free system, all CPUs can continue doing other useful work without stalls. Lock-free algorithms increase the overall throughput of a system by occassionally increasing the latency of a particular transaction. Most high-end database systems are based on lock-free algorithms, to varying degrees.
By contrast, wait-free algorithms ensure that in addition to all CPUs continuing to do useful work, no computation can ever be blocked by another computation. Wait-free algorithms have stronger guarantees than lock-free algorithms, and ensure a high thorughput without sacrificing latency of a particular transaction. They’re also much harder to implement, test, and debug. The lockless page cache patches to the Linux kernel are an example of a wait-free system.
In a situation where a system handles dozens of concurrent transactions and has soft latency requirements, lock-free systems are a good compromise between development complexity and high concurrency requirements. A database server for a website is a good candidate for a lock-free design. While any given transaction might block, there are always more transactions to process in the meantime, so the CPUs will never stay idle. The challenge is to build a transaction scheduler that maintains a good mean latency, and a well bounded standard deviation.
In a scenario where a system has roughly as many concurrent transactions as CPU cores, or has hard real-time requirements, the developers need to spend the extra time to build wait-free systems. In these cases blocking a single transaction isn’t acceptable – either because there are no other transactions for the CPUs to handle, minimizing the throughput, or a given transaction needs to complete with a well defined non-probabilistic time period. Nuclear reactor control software is a good candidate for wait-free systems.
RethinkDB is a lock-free system. On a machine with N CPU cores, under most common workloads, we can gurantee that no core will stay idle and no IO pipeline capacity is wasted as long as there are roughly N * 4 concurrent transactions. For example, on an eight core system, no piece of hardware will sit idle if RethinkDB is handling roughly 32 concurrent transactions or more. If there are fewer than 32 transactions, you’ve likely overpaid for some of the cores. (Of course if you only have 32 concurrent transactions, you don’t need an eight-core machine).
Interested in working at RethinkDB? We’re hiring – please see our jobs page for more details.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 15:10:33 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:1:{i:0;a:5:{s:4:"data";s:13:"Announcements";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:3450:"
There are two types of non-blocking thread synchronization algorithms – lock-free, and wait-free. Their meaning is often confused. In lock-free systems, while any particular computation may be blocked for some period of time, all CPUs are able to continue performing other computations. To put it differently, while a given thread might be blocked by other threads in a lock-free system, all CPUs can continue doing other useful work without stalls. Lock-free algorithms increase the overall throughput of a system by occassionally increasing the latency of a particular transaction. Most high-end database systems are based on lock-free algorithms, to varying degrees.


By contrast, wait-free algorithms ensure that in addition to all CPUs continuing to do useful work, no computation can ever be blocked by another computation. Wait-free algorithms have stronger guarantees than lock-free algorithms, and ensure a high thorughput without sacrificing latency of a particular transaction. They’re also much harder to implement, test, and debug. The lockless page cache patches to the Linux kernel are an example of a wait-free system.


In a situation where a system handles dozens of concurrent transactions and has soft latency requirements, lock-free systems are a good compromise between development complexity and high concurrency requirements. A database server for a website is a good candidate for a lock-free design. While any given transaction might block, there are always more transactions to process in the meantime, so the CPUs will never stay idle. The challenge is to build a transaction scheduler that maintains a good mean latency, and a well bounded standard deviation.


In a scenario where a system has roughly as many concurrent transactions as CPU cores, or has hard real-time requirements, the developers need to spend the extra time to build wait-free systems. In these cases blocking a single transaction isn’t acceptable – either because there are no other transactions for the CPUs to handle, minimizing the throughput, or a given transaction needs to complete with a well defined non-probabilistic time period. Nuclear reactor control software is a good candidate for wait-free systems.


RethinkDB is a lock-free system. On a machine with N CPU cores, under most common workloads, we can gurantee that no core will stay idle and no IO pipeline capacity is wasted as long as there are roughly N * 4 concurrent transactions. For example, on an eight core system, no piece of hardware will sit idle if RethinkDB is handling roughly 32 concurrent transactions or more. If there are fewer than 32 transactions, you’ve likely overpaid for some of the cores. (Of course if you only have 32 concurrent transactions, you don’t need an eight-core machine).


Interested in working at RethinkDB? We’re hiring – please see our jobs page for more details.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:15:"Slava Akhmechet";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:20;a:6:{s:4:"data";s:98:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:37:"eReaders and the Danger of Price Wars";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:60:"tag:typepad.com,2003:post-6a00d83452e46469e20133f1a6ead8970b";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:58:"http://www.theopenforce.com/2010/06/ereader-price-war.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:3183:"
  A longer version of this story is published at www.opensources.comLast week, Barnes & Noble announced they would cut the price on their wireless Nook eReader, from $259 to $199 ($149 for a new WiFi-only edition.)  Many thought this was a good opportunity for the third place contender to gain market share.  But within a few hours Amazon beat Barnes & Noble's price by $10, marking down the Kindle 2 to a mere $189.

As the New York Times notes:



	The price cuts were made as manufacturers of e-readers faced a mounting threat from Apple’s iPad. Even though it is far more expensive than the e-readers, the iPad, which starts at $500, performs a range of functions with a versatile, colorful display that contrasts sharply with the static, monochrome screen of e-book readers. Apple said it sold more than two million iPads in the two months since the tablet’s introduction... Analysts had expected the prices of e-readers would gradually fall because of the natural decline in component costs and the increased profitability of e-books themselves.

	The price cuts should add further momentum to what, despite incursions by the iPad, has been a growing market for dedicated e-reading devices. Amazon and its rivals are on pace to sell 6.6 million e-reading devices this year, up from 3.1 million in 2009, according to Forrester.

If Amazon, Barnes & Noble, Sony et al manage to sell 6 million eReaders this year, that would be impressive growth for a category that has been lackluster to date.  Amazon has never broken out sales of it's Kindle line, but by all appearances it's the leading standalone eReader and likely has sold a couple of million units in its three year history.

In comparison, Apple has sold more than 3 million iPads in its first 80 days. And they're expanding into 9 more countries next month.  Analysts are predicting that the iPad could sell between 5 and 10 million units this year, which blows Amazon's Kindle out of the water.  And unlike Amazon, Apple actually makes money with it's iPad since it's costs are around $260 for the $499 entry level product and margins improve on the higher end units.  

But its worth considering a few questions:




Will price cuts making any difference competing against the iPad?
Or does it just increase the burn of a money-losing business?
Why is Apple's iPad business profitable and Amazon's Kindle isn't?
If you could chose to be in either business, which would you choose?
And what does all this have to do with open source?

The key point here is that price is just one part of a disruptive strategy.  No doubt, part of the success of MySQL, Red Hat, jBoss, Alfresco, Zimbra, Pentaho, Revolution Analytics et al, comes from delivering 90% of the benefit for 10% of the price of incumbents. The trick is do to do in a manner that is profitable but that incumbents cannot respond to because of their higher cost of operations.  (And remember, most open source users don't pay anything!)  Read a longer version of this story at www.opensources.com
Amazon: Kindle $189

New York Times: In Price Wars, eReaders go below $200

";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 14:25:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:12:{i:0;a:5:{s:4:"data";s:8:"Business";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:13:"Hot Companies";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:13:"Amazon Kindle";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:10:"Apple iPad";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:18:"Barnes & Noble";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:5:"cloud";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:10:"disruption";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:8:"freemium";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:9;a:5:{s:4:"data";s:4:"Nook";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:10;a:5:{s:4:"data";s:11:"open source";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:11;a:5:{s:4:"data";s:5:"price";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:5938:"

Newsweek_ipad 
 
A longer version of this story is published at www.opensources.com
Last week, Barnes & Noble announced they would cut the price on their wireless Nook eReader, from $259 to $199 ($149 for a new WiFi-only edition.)  Many thought this was a good opportunity for the third place contender to gain market share.  But within a few hours Amazon beat Barnes & Noble's price by $10, marking down the Kindle 2 to a mere $189.



As the New York Times notes:






	The price cuts were made as manufacturers of e-readers faced a mounting threat from Apple’s iPad. Even though it is far more expensive than the e-readers, the iPad, which starts at $500, performs a range of functions with a versatile, colorful display that contrasts sharply with the static, monochrome screen of e-book readers. Apple said it sold more than two million iPads in the two months since the tablet’s introduction... Analysts had expected the prices of e-readers would gradually fall because of the natural decline in component costs and the increased profitability of e-books themselves.



	The price cuts should add further momentum to what, despite incursions by the iPad, has been a growing market for dedicated e-reading devices. Amazon and its rivals are on pace to sell 6.6 million e-reading devices this year, up from 3.1 million in 2009, according to Forrester.



If Amazon, Barnes & Noble, Sony et al manage to sell 6 million eReaders this year, that would be impressive growth for a category that has been lackluster to date.  Amazon has never broken out sales of it's Kindle line, but by all appearances it's the leading standalone eReader and likely has sold a couple of million units in its three year history.



In comparison, Apple has sold more than 3 million iPads in its first 80 days. And they're expanding into 9 more countries next month.  Analysts are predicting that the iPad could sell between 5 and 10 million units this year, which blows Amazon's Kindle out of the water.  And unlike Amazon, Apple actually makes money with it's iPad since it's costs are around $260 for the $499 entry level product and margins improve on the higher end units.  



But its worth considering a few questions:







The key point here is that price is just one part of a disruptive strategy.  No doubt, part of the success of MySQL, Red Hat, jBoss, Alfresco, Zimbra, Pentaho, Revolution Analytics et al, comes from delivering 90% of the benefit for 10% of the price of incumbents. The trick is do to do in a manner that is profitable but that incumbents cannot respond to because of their higher cost of operations.  (And remember, most open source users don't pay anything!)  
Read a longer version of this story at www.opensources.com



PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Zack Urlocker";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:21;a:6:{s:4:"data";s:98:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:39:"MySQL’s SQL Deviations and Extensions";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:36:"http://www.pythian.com/news/?p=13845";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:71:"http://www.pythian.com/news/13845/mysqls-sql-deviations-and-extensions/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:1121:"Today at Kaleidoscope I will be doing a 90-minute session comparing MySQL’s SQL syntax to the ANSI/ISO SQL:2003 standard, entitled What Do You Mean, “SQL Syntax Error”?
You can download the PDF slides now.

For those that may be following along the presentation later today (4 pm Eastern time), here are some links that I may throw out during the session:
SQL 2003 standard – actually it is “Information taken from the Final Committee Draft (FCD) of ISO/IEC 9075-2:2003″ but it’s extremely close to the actual standard.  The actual standard is a document that costs a non-trivial amount of money to get, and cannot be republished.
The MySQL Administrator’s Bible, from which this content was taken from.
The MySQL Manual page for sql_mode values.
Information about and examples of quoting in MySQL
The MySQL Manual page for CREATE TABLE which contains all the special extensions MySQL has for tables.
The MySQL Manual page for PROCEDURE ANALYSE().
The MySQL Manual page for the SHOW extension
PDF slides for an EXPLAIN presentation I have done, and the EXPLAIN Cheatsheet.
";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 13:53:22 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:12:{i:0;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:6:"Oracle";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:19:"Pythian Appearances";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:14:"Technical Blog";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:12:"CREATE TABLE";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:7:"explain";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:12:"kaleidoscope";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:5:"odtug";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:7:"Pythian";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:9;a:5:{s:4:"data";s:3:"sql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:10;a:5:{s:4:"data";s:12:"SQL standard";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:11;a:5:{s:4:"data";s:8:"SQL_MODE";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:2208:"
Today at Kaleidoscope I will be doing a 90-minute session comparing MySQL’s SQL syntax to the ANSI/ISO SQL:2003 standard, entitled What Do You Mean, “SQL Syntax Error”?


You can download the PDF slides now.



For those that may be following along the presentation later today (4 pm Eastern time), here are some links that I may throw out during the session:



PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:16:"Sheeri K. Cabral";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:22;a:6:{s:4:"data";s:48:"
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:23:"Velocity 2010 In Review";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:49:"http://brian.moonspot.net/velocity-2010-in-review";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:49:"http://brian.moonspot.net/velocity-2010-in-review";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:3096:"I just got back from Velocity for the third straight year. I have been to all three of them which is kind of a neat little club to be in. The first one only had maybe 300 people. This year there were over 1,000 attendees. Registration was shut down by the fire code for the rooms we were using. Most sessions had standing room only. It was awesome.The people that talk at Velocity are really smart. I am always humbled by the likes of John Allspaw. He and I see eye to eye on a lot, but he is so much better at explaining to people and showing them how to make the ideas work. I wish I had his charisma when at the podium. I was lucky enough to write a chapter in a book for John this year. He and Velocity co-chairperson Jesse Robbins organized and authored a book titled Web Operations that debuted at the conference. I basically just told and expanded on my Yahoo story. John loves that story for some reason. I was happy to be a part of it. So many smart people in that book.The IE9 technology preview dropped while we were there. HTML 5, CSS 3 and more in there. One feature where Microsoft is actually ahead of the curve is in a new DOM level measurement feature. Basically they expose statistics via the DOM about the time it takes to do different things in the page. The other browser vendors in attendance (Google and Mozilla) vowed to support the same data. Another big advancement of IE9 is the heavy use of the GPU for rendering the pages. They have a real advantage here. They are the only browser vendor that is now locked to one operating system. IE9 will require Vista or higher. They can really max out the system for faster rendering.As usual some of the best content was in the hall ways and bar. We hung out with Theo Schlossnagle from OmniTI and talked about Reconnoiter. It is a kind of Cacti/Ganglia/Nagios all in one. I got to see the Six Apart guys again this year. That is becoming an annual thing. I shared our new Gearman assisted proxy with them. They do some similar stuff for TypePad. More on that proxy later this year. I met a guy from CloudTest. It sounds like a really good use of on-demand cloud resources. I am gonna talk with them about some possible testing.Membase also dropped while we were there. Most of the persistent key/value stores I have used have disappointed me or just been way too complex for our needs. We don't want a memcached replacement. It does its job damn well. I just need a place to store adhoc data for various applications. Membase is promising because the guys that wrote it are core memcached contributors. There is a company behind it, so it is not as inviting as Drizzle. But, the code is on GitHub so it is more open than say MySQL. Time will tell.If you have not been to Velocity I encourage you to go next year. It is right for all types of people in the web business. Developers can learn about performance in new ways that will change they way they write code. Operations can learn techniques to make their work day much less painful. Everyone will learn how to empower their business to achieve the goals of the business. ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 13:00:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:2:{i:0;a:5:{s:4:"data";s:6:"devops";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:11:"scalability";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:3907:"I just got back from Velocity for the third straight year. I have been to all three of them which is kind of a neat little club to be in. The first one only had maybe 300 people. This year there were over 1,000 attendees. Registration was shut down by the fire code for the rooms we were using. Most sessions had standing room only. It was awesome.

The people that talk at Velocity are really smart. I am always humbled by the likes of John Allspaw. He and I see eye to eye on a lot, but he is so much better at explaining to people and showing them how to make the ideas work. I wish I had his charisma when at the podium. I was lucky enough to write a chapter in a book for John this year. He and Velocity co-chairperson Jesse Robbins organized and authored a book titled Web Operations that debuted at the conference. I basically just told and expanded on my Yahoo story. John loves that story for some reason. I was happy to be a part of it. So many smart people in that book.

The IE9 technology preview dropped while we were there. HTML 5, CSS 3 and more in there. One feature where Microsoft is actually ahead of the curve is in a new DOM level measurement feature. Basically they expose statistics via the DOM about the time it takes to do different things in the page. The other browser vendors in attendance (Google and Mozilla) vowed to support the same data. Another big advancement of IE9 is the heavy use of the GPU for rendering the pages. They have a real advantage here. They are the only browser vendor that is now locked to one operating system. IE9 will require Vista or higher. They can really max out the system for faster rendering.

As usual some of the best content was in the hall ways and bar. We hung out with Theo Schlossnagle from OmniTI and talked about Reconnoiter. It is a kind of Cacti/Ganglia/Nagios all in one. I got to see the Six Apart guys again this year. That is becoming an annual thing. I shared our new Gearman assisted proxy with them. They do some similar stuff for TypePad. More on that proxy later this year. I met a guy from CloudTest. It sounds like a really good use of on-demand cloud resources. I am gonna talk with them about some possible testing.

Membase also dropped while we were there. Most of the persistent key/value stores I have used have disappointed me or just been way too complex for our needs. We don't want a memcached replacement. It does its job damn well. I just need a place to store adhoc data for various applications. Membase is promising because the guys that wrote it are core memcached contributors. There is a company behind it, so it is not as inviting as Drizzle. But, the code is on GitHub so it is more open than say MySQL. Time will tell.

If you have not been to Velocity I encourage you to go next year. It is right for all types of people in the web business. Developers can learn about performance in new ways that will change they way they write code. Operations can learn techniques to make their work day much less painful. Everyone will learn how to empower their business to achieve the goals of the business. 

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:10:"Brian Moon";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:23;a:6:{s:4:"data";s:58:"
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:31:"Using Syslog with MySQL Cluster";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:32:"http://www.clusterdb.com/?p=1141";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:63:"http://www.clusterdb.com/mysql/using-syslog-with-mysql-cluster/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:1725:"By default, MySQL Cluster sends log data to a file but you can also send it to the console or to Syslog; this article explains how to send it to Syslog. The example given here is for LINUX.
In this example, I’ll use the “user” syslog facility name and so the first step is to make sure that syslog is configured to route those messages. If this hasn’t already been configured then add the following lines to /etc/rsyslog.conf:
# Log user messages to local files
user.*    /var/log/user
For the changes to take effect, restart the syslog service:
[root@ws1 etc]# service rsyslog restart
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]
Note that you should make those changes as root.
Still as root, start up a stream of  any additions to the new log file:
[root@ws1 etc]# tail -f /var/log/user
To tell Cluster to use Syslog, add this line into the [ndb_mgmd] section in config.ini:
LogDestination=SYSLOG:facility=user
and then start up your Cluster as normal.
You should now be able to see that MySQL Cluster information is being logged to /var/log/user.
You can adjust how much information is logged either through the config file or from the ndb_mgm tool, for example – to see when global checkpoints are written:
ndb_mgm> all clusterlog checkpoint=15
Executing CLUSTERLOG CHECKPOINT=15 on node 3 OK!
Executing CLUSTERLOG CHECKPOINT=15 on node 4 OK!
Note that a log-level of 15 will show all logs and 0 will show none. Other log categories besides CHECKPOINT are STARTUP, SHUTDOWN, STATISTICS, NODERESTART, CONNECTION, INFO, ERROR, CONGESTION, DEBUG and BACKUP.
";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 12:40:44 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:13:"MySQL Cluster";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:17:"MySQL Cluster CGE";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:6:"syslog";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:2260:"
By default, MySQL Cluster sends log data to a file but you can also send it to the console or to Syslog; this article explains how to send it to Syslog. The example given here is for LINUX.


In this example, I’ll use the “user” syslog facility name and so the first step is to make sure that syslog is configured to route those messages. If this hasn’t already been configured then add the following lines to /etc/rsyslog.conf:


# Log user messages to local files
user.*    /var/log/user


For the changes to take effect, restart the syslog service:


[root@ws1 etc]# service rsyslog restart
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]


Note that you should make those changes as root.


Still as root, start up a stream of  any additions to the new log file:


[root@ws1 etc]# tail -f /var/log/user


To tell Cluster to use Syslog, add this line into the [ndb_mgmd] section in config.ini:


LogDestination=SYSLOG:facility=user


and then start up your Cluster as normal.


You should now be able to see that MySQL Cluster information is being logged to /var/log/user.


You can adjust how much information is logged either through the config file or from the ndb_mgm tool, for example – to see when global checkpoints are written:


ndb_mgm> all clusterlog checkpoint=15
Executing CLUSTERLOG CHECKPOINT=15 on node 3 OK!
Executing CLUSTERLOG CHECKPOINT=15 on node 4 OK!


Note that a log-level of 15 will show all logs and 0 will show none. Other log categories besides CHECKPOINT are STARTUP, SHUTDOWN, STATISTICS, NODERESTART, CONNECTION, INFO, ERROR, CONGESTION, DEBUG and BACKUP.



PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Andrew Morgan";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:24;a:6:{s:4:"data";s:58:"
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:52:"MySQL Cluster presentation at Oracle Open World 2010";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:32:"http://www.clusterdb.com/?p=1136";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:84:"http://www.clusterdb.com/mysql/mysql-cluster-presentation-at-oracle-open-world-2010/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:674:"As part of “MySQL Sunday” at this year’s Oracle Open World, Mat Keep and I will be presenting on the latest MySQL Cluster features. We’ll be presenting at 15:30 (Pacific Time) on 19th September (the event starts with a key note at 12:30).
If you’re attending Oracle Open World then please indicate that you’d like to attend the MySQL Sunday when you register. If you aren’t planning to go to Oracle Open World but will be in the San Francisco area then buying a Discover pass (only $50 if you register by 16 July) will get you into the MySQL Sunday sessions. Register here.
For details on the presentations and speakers, check here.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 10:16:36 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:13:"MySQL Cluster";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:17:"MySQL Cluster 7.1";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:17:"MySQL Cluster CGE";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1355:"
As part of “MySQL Sunday” at this year’s Oracle Open World, Mat Keep and I will be presenting on the latest MySQL Cluster features. We’ll be presenting at 15:30 (Pacific Time) on 19th September (the event starts with a key note at 12:30).


If you’re attending Oracle Open World then please indicate that you’d like to attend the MySQL Sunday when you register. If you aren’t planning to go to Oracle Open World but will be in the San Francisco area then buying a Discover pass (only $50 if you register by 16 July) will get you into the MySQL Sunday sessions. Register here.


For details on the presentations and speakers, check here.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Andrew Morgan";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:25;a:6:{s:4:"data";s:38:"
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:25:"1.5.1 RC 2 Now Available!";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:61:"http://infinidb.org/infinidb-blog/151-rc-2-now-available.html";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:61:"http://infinidb.org/infinidb-blog/151-rc-2-now-available.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:501:"We are pleased to announce the availability of the 1.5.1 RC 2 release of InfiniDB Community Edition.  We plan on this being our last release candidate before our 1.5 Final release in early July.  This release includes a number of bug fixes that you can see at http://bugs.launchpad.net/infinidb.  You can download the latest InfiniDB binaries, source code, and updated documentation at: http://infinidb.org/downloads.  We welcome your feedback&Read More...";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Mon, 28 Jun 2010 01:05:58 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:729:"
We are pleased to announce the availability of the 1.5.1 RC 2 release of InfiniDB Community Edition.  We plan on this being our last release candidate before our 1.5 Final release in early July.  

This release includes a number of bug fixes that you can see at http://bugs.launchpad.net/infinidb.  You can download the latest InfiniDB binaries, source code, and updated documentation at: http://infinidb.org/downloads.  We welcome your feedback&Read More...
PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:31:"Calpont & InfiniDB Project Team";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:26;a:6:{s:4:"data";s:43:"
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:65:"Running MySQL Cluster without arbitrator: what it's really about.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:25:"288 at http://openlife.cc";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:97:"http://openlife.cc/blogs/2010/june/running-mysql-cluster-without-arbitrator-what-its-really-about";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:626:"Geert made us aware that MySQL Cluster now provides the possibility to disable arbitration in order to use an external arbitration mechanism. This is a really important feature, because... well, not really, but only because I was the one who designed it :-)
Coming up with the concept and the two parameters Arbitration=WaitExternal and ArbitrationTimeout=n took a few weeks of discussion. Once we agreed on how to do it, I think Jonas coded it in 20 minutes on the mezzanine floor of the Hyatt, Santa Clara. After that MySQL conference I soon resigned from Sun, so I had now idea what then happened to this feature.
read more";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Sat, 26 Jun 2010 21:00:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:1:{i:0;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1092:"
Geert made us aware that MySQL Cluster now provides the possibility to disable arbitration in order to use an external arbitration mechanism. This is a really important feature, because... well, not really, but only because I was the one who designed it :-)


Coming up with the concept and the two parameters Arbitration=WaitExternal and ArbitrationTimeout=n took a few weeks of discussion. Once we agreed on how to do it, I think Jonas coded it in 20 minutes on the mezzanine floor of the Hyatt, Santa Clara. After that MySQL conference I soon resigned from Sun, so I had now idea what then happened to this feature.


read more

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:11:"Henrik Ingo";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:27;a:6:{s:4:"data";s:48:"
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:35:"[MySQL][Spider]Spider-2.21 released";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:70:"tag:blogger.com,1999:blog-7870178081855084823.post-2659098112625419729";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:75:"http://wild-growth.blogspot.com/2010/06/mysqlspiderspider-221-released.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:979:"I'm pleased to announce the release of Spider storage engine version 2.21(beta).Spider is a Storage Engine for database sharding.http://spiderformysql.com/The main changes in this version are following.- Add table parameter "bka_mode" and "bka_engine".- Add server parameter "spider_bka_mode" and "spider_bka_engine".  This parameters are for improvement performance for using BKA and using a lot of conditions with IN clause and so on."mysql-5.1.44.slave-trx-retry.diff" is included in the source tar ball from this release.mysql-5.1.44.slave-trx-retry.diff adds a option "slave_transaction_retry_errors". This parameter is used like "slave_transaction_retry_errors=1158,1159,2013,12701". Slave sql thread retries the transaction if slave sql thread encounters the errors at this parameter. Number of retries are at "slave_transaction_retries".Please see "99_change_logs.txt" in the download documents for checking other changes.Thanks to Mikiya for your advice.Enjoy!";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Sat, 26 Jun 2010 17:19:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:2:{i:0;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:6:"Spider";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1447:"I'm pleased to announce the release of Spider storage engine version 2.21(beta).
Spider is a Storage Engine for database sharding.
http://spiderformysql.com/

The main changes in this version are following.
- Add table parameter "bka_mode" and "bka_engine".
- Add server parameter "spider_bka_mode" and "spider_bka_engine".
  This parameters are for improvement performance for using BKA and using a lot of conditions with IN clause and so on.

"mysql-5.1.44.slave-trx-retry.diff" is included in the source tar ball from this release.
mysql-5.1.44.slave-trx-retry.diff adds a option "slave_transaction_retry_errors". This parameter is used like "slave_transaction_retry_errors=1158,1159,2013,12701". Slave sql thread retries the transaction if slave sql thread encounters the errors at this parameter. Number of retries are at "slave_transaction_retries".

Please see "99_change_logs.txt" in the download documents for checking other changes.
Thanks to Mikiya for your advice.

Enjoy!

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Kentoku SHIBA";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:28;a:6:{s:4:"data";s:58:"
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:61:"Ever tried calling a win32ole (COM) object from Ruby’s DRb?";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:32:"http://www.linuxbloke.com/?p=266";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:32:"http://www.linuxbloke.com/?p=266";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:8988:"Before we get started here, let me state that I am using Ruby 1.9.1 (I refuse to look back!), and that I have not tested this solution on Ruby 1.8.6, but it should work there as well, though I may have some 1.9-isms in my code. Should be easy enough to spot.
I am working on writing an application in Ruby that can talk to an Windows application that has an ActiveX COM Automation object exposed. Ruby is basically the wrapper so that I can access the application from the Linux side of the world. So, I am using Ruby’s DRb to bridge those worlds because, after all, I am the Linux Bloke!
Well, as you may have guessed, I ran into problems with this approach. I simply could not call the COM objects from a call initiated with DRb, though I could call them directly just fine. After scratching my head a bit, I figured it out.
The win32ole module that runs on the Windows side of the world in Ruby only wants to run in the same thread that it was started in. win32ole is simply not thread-safe, and this has to do in large part to how ActiveX works under Windows. No need to delve into the gory details as we want code that works already!
DRb is very much all about threads. The DRb Server runs in a separate thread, and threads are launched each time a DRb request comes in. Threads abound like crazy! After all, it is very clear that the implementation of DRb was based, in part, on the Java threading model and Java’s RMI. But we knew that. We know that Ruby Threads parrot Java Threads. And I’ve done a lot of work with Java Threads in the past and almost feel a bit of “déjà vu” in working with them in Ruby. Oh the days…
But I digress.
We have a major problem here. How do we get around it, without having to throw out DRb and doing something funky like writing some custom RPC bit just to make Windows happy?
Well, as you may have guess, the Linux Bloke created the very solution you need!! Funnel!
Funnel works by wrapping a given object with a “meta” object that can then be called from any thread. All the calls are actually queued up and processed by the thread the target object wants to run in. The calling threads block until the target object returns the call, and the result objects are stuffed somewhere so that the calling thread can find them.
It’s all very transparent and you need not do anything special — much. You will need to call process_funnel_messages() in the funneled thread. And you may do this once in which case process_funnel_messages() will loop forever and never return, or you can call it at regular intervals if you need to do other processing in that same thread.
You, of course, can use Funnel anywhere you need to funnel calls from multiple threads to a single thread to access something that is not inherently thread-safe or thread-aware.
The downloadable code is posted here:

?Download funnel.rb1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
=begin rdoc
Funnel created by Fred Mitchell (LinuxBloke.com) on 2010-06-05                         
 
=Funnel -- funnel calls to an object to a specific thread that created said object.    
 
With some systems, like win32ole, the system basically wants to run on the same thread
the system was started on. To facilitate that need in a multi-threaded environment,
we create the Funnel.                                                                  
 
The Funnel wrapper on an object will basically intercept all method calls and
funnel those calls to the wrapped object in the thread it was created in. The
caller thread will basically block until the Funnel calls the target object's method
and will be given, as a return, the result object of that call.                        
 
The Funnel thread will basically sit in a loop waiting for something to come in,
and wake up to process the entries, then go back to sleep until the next ones come
in.                                                                                    
 
Any exceptions (or errors) that occur in the Funnel shall be
thrown to the caller thread, as though the exception took place in that thread.        
 
This code is released under the GPLv3.                                                 
 
=end                         
 
module Funnel
  class Wrapper
    def initialize(target)
      @targetOb = target
      @targetThr = Thread.current
      @targetThr[:methQueue] = [] if @targetThr[:methQueue].nil?
    end                                                                                
 
    def method_missing(meth, *parms)
      Thread.current[:methResult] = :nothing_yet
      @targetThr[:methQueue] &lt;&lt; [@targetOb, meth, Thread.current, parms]               
 
      # Thing is, we may have gotten a response already!
      while Thread.current[:methResult] == :nothing_yet
        if @targetThr.stop?
          @targetThr.wakeup
          # Thread.stop
        end
        Thread.pass
      end
      Thread.current[:methResult]
    end
  end                                                                                  
 
  # Called by the orginal thread to process object messages.
  # This function never returns.
  def process_funnel_messages(loop_forever = true)
    begin
      meth = nil
      (ob, meth, thr, parms) = Thread.current[:methQueue].shift unless Thread.current[\
:methQueue].nil?
      unless meth.nil?
        begin
          thr[:methResult] = ob.send(meth, *parms)
          thr.run
        rescue
          thr.raise($!)
        end
      else
        Thread.stop if loop_forever
      end
    end while loop_forever
  end                                                                                  
 
  def wrap(target)
    Wrapper.new(target)
  end
end

And here is an example of its use:

?Download example_of_funnel_use.rb1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
require 'funnel'                                                                       
include Funnel                                                                         
 
class StupidThreadUnsafeThing                                                          
  def callme                                                                           
    puts "*** I've been called. My thread is"                                          
    p Thread.current                                                                   
    puts                                                                               
  end                                                                                  
end                                                                                    
 
stut = StupidThreadUnsafeThing.new                                                     
 
# This is the easy to use wrapper                                                      
fstut = wrap stut                                                                      
 
stut.callme                                                                            
 
Thread.new do                                                                          
  10.times do |i|                                                                      
    sleep 1                                                                            
    Thread.new {                                                                       
      puts "XXX #{i} calling stut from thread"                                         
      p Thread.current                                                                 
      fstut.callme                                                                     
    }                                                                                  
  end                                                                                  
  exit                                                                                 
end                                                                                    
 
# Here we loop forever processing messages.                                            
# Optionally, we could call this repeateady                                            
# to process messages by using a parameter of                                          
# "false".                                                                             
process_funnel_messages

This code is fairly straightforward, as you can see. If there is enough interest, I’ll consider turning this into a gem. ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Sat, 26 Jun 2010 14:25:31 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:11:"Development";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:4:"Ruby";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:7:"Windows";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:5:"linux";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:12250:"
Before we get started here, let me state that I am using Ruby 1.9.1 (I refuse to look back!), and that I have not tested this solution on Ruby 1.8.6, but it should work there as well, though I may have some 1.9-isms in my code. Should be easy enough to spot.


I am working on writing an application in Ruby that can talk to an Windows application that has an ActiveX COM Automation object exposed. Ruby is basically the wrapper so that I can access the application from the Linux side of the world. So, I am using Ruby’s DRb to bridge those worlds because, after all, I am the Linux Bloke!


Well, as you may have guessed, I ran into problems with this approach. I simply could not call the COM objects from a call initiated with DRb, though I could call them directly just fine. After scratching my head a bit, I figured it out.


The win32ole module that runs on the Windows side of the world in Ruby only wants to run in the same thread that it was started in. win32ole is simply not thread-safe, and this has to do in large part to how ActiveX works under Windows. No need to delve into the gory details as we want code that works already!


DRb is very much all about threads. The DRb Server runs in a separate thread, and threads are launched each time a DRb request comes in. Threads abound like crazy! After all, it is very clear that the implementation of DRb was based, in part, on the Java threading model and Java’s RMI. But we knew that. We know that Ruby Threads parrot Java Threads. And I’ve done a lot of work with Java Threads in the past and almost feel a bit of “déjà vu” in working with them in Ruby. Oh the days…


But I digress.


We have a major problem here. How do we get around it, without having to throw out DRb and doing something funky like writing some custom RPC bit just to make Windows happy?


Well, as you may have guess, the Linux Bloke created the very solution you need!! Funnel!


Funnel works by wrapping a given object with a “meta” object that can then be called from any thread. All the calls are actually queued up and processed by the thread the target object wants to run in. The calling threads block until the target object returns the call, and the result objects are stuffed somewhere so that the calling thread can find them.


It’s all very transparent and you need not do anything special — much. You will need to call process_funnel_messages() in the funneled thread. And you may do this once in which case process_funnel_messages() will loop forever and never return, or you can call it at regular intervals if you need to do other processing in that same thread.


You, of course, can use Funnel anywhere you need to funnel calls from multiple threads to a single thread to access something that is not inherently thread-safe or thread-aware.


The downloadable code is posted here:



?Download funnel.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

=begin rdoc
Funnel created by Fred Mitchell (LinuxBloke.com) on 2010-06-05                         
 
=Funnel -- funnel calls to an object to a specific thread that created said object.    
 
With some systems, like win32ole, the system basically wants to run on the same thread
the system was started on. To facilitate that need in a multi-threaded environment,
we create the Funnel.                                                                  
 
The Funnel wrapper on an object will basically intercept all method calls and
funnel those calls to the wrapped object in the thread it was created in. The
caller thread will basically block until the Funnel calls the target object's method
and will be given, as a return, the result object of that call.                        
 
The Funnel thread will basically sit in a loop waiting for something to come in,
and wake up to process the entries, then go back to sleep until the next ones come
in.                                                                                    
 
Any exceptions (or errors) that occur in the Funnel shall be
thrown to the caller thread, as though the exception took place in that thread.        
 
This code is released under the GPLv3.                                                 
 
=end                         
 
module Funnel
  class Wrapper
    def initialize(target)
      @targetOb = target
      @targetThr = Thread.current
      @targetThr[:methQueue] = [] if @targetThr[:methQueue].nil?
    end                                                                                
 
    def method_missing(meth, *parms)
      Thread.current[:methResult] = :nothing_yet
      @targetThr[:methQueue] &lt;&lt; [@targetOb, meth, Thread.current, parms]               
 
      # Thing is, we may have gotten a response already!
      while Thread.current[:methResult] == :nothing_yet
        if @targetThr.stop?
          @targetThr.wakeup
          # Thread.stop
        end
        Thread.pass
      end
      Thread.current[:methResult]
    end
  end                                                                                  
 
  # Called by the orginal thread to process object messages.
  # This function never returns.
  def process_funnel_messages(loop_forever = true)
    begin
      meth = nil
      (ob, meth, thr, parms) = Thread.current[:methQueue].shift unless Thread.current[\
:methQueue].nil?
      unless meth.nil?
        begin
          thr[:methResult] = ob.send(meth, *parms)
          thr.run
        rescue
          thr.raise($!)
        end
      else
        Thread.stop if loop_forever
      end
    end while loop_forever
  end                                                                                  
 
  def wrap(target)
    Wrapper.new(target)
  end
end



And here is an example of its use:



?Download example_of_funnel_use.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

require 'funnel'                                                                       
include Funnel                                                                         
 
class StupidThreadUnsafeThing                                                          
  def callme                                                                           
    puts "*** I've been called. My thread is"                                          
    p Thread.current                                                                   
    puts                                                                               
  end                                                                                  
end                                                                                    
 
stut = StupidThreadUnsafeThing.new                                                     
 
# This is the easy to use wrapper                                                      
fstut = wrap stut                                                                      
 
stut.callme                                                                            
 
Thread.new do                                                                          
  10.times do |i|                                                                      
    sleep 1                                                                            
    Thread.new {                                                                       
      puts "XXX #{i} calling stut from thread"                                         
      p Thread.current                                                                 
      fstut.callme                                                                     
    }                                                                                  
  end                                                                                  
  exit                                                                                 
end                                                                                    
 
# Here we loop forever processing messages.                                            
# Optionally, we could call this repeateady                                            
# to process messages by using a parameter of                                          
# "false".                                                                             
process_funnel_messages



This code is fairly straightforward, as you can see. If there is enough interest, I’ll consider turning this into a gem. 

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Fred Mitchell";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:29;a:6:{s:4:"data";s:63:"
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:113:"Webinar today – Scaling Web Services with MySQL Cluster, Part 1: An Alternative to MySQL Server & memcached";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:32:"http://www.clusterdb.com/?p=1097";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:141:"http://www.clusterdb.com/mysql-cluster/webinar-today-scaling-web-services-with-mysql-cluster-part-1-an-alternative-to-mysql-server-memcached/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:1119:"The replay of this webinar is now available from http://www.mysql.com/news-and-events/on-demand-webinars/display-od-545.html
MySQL and memcached has become, and will remain, the foundation for many dynamic web services with proven deployments in some of the largest and most prolific names on the web. There are classes of web services however that are update-intensive, demanding real-time responsiveness and continuous availability. In these cases, MySQL Cluster provides the familiarity and ease-of-use of the regular MySQL Server, while delivering significantly higher levels of write performance with less complexity, lower latency and 99.999% availability. This webinar will discuss the use-cases for both approaches, and provide an insight into how MySQL Cluster is enabling users to scale their update-intensive web services.
The webinar starts at 09:00 Pacific/17:00 UK/18:00 CET today (June 9th 2010).
Still time to register (for free) at http://www.mysql.com/news-and-events/web-seminars/display-545.html – even if you can’t attend, this way you’ll get sent a link to the charts and replay.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Sat, 26 Jun 2010 10:05:42 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:5:{i:0;a:5:{s:4:"data";s:13:"MySQL Cluster";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:9:"Memcached";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:3:"web";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:7:"webinar";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1563:"
The replay of this webinar is now available from http://www.mysql.com/news-and-events/on-demand-webinars/display-od-545.html


MySQL and memcached has become, and will remain, the foundation for many dynamic web services with proven deployments in some of the largest and most prolific names on the web. There are classes of web services however that are update-intensive, demanding real-time responsiveness and continuous availability. In these cases, MySQL Cluster provides the familiarity and ease-of-use of the regular MySQL Server, while delivering significantly higher levels of write performance with less complexity, lower latency and 99.999% availability. This webinar will discuss the use-cases for both approaches, and provide an insight into how MySQL Cluster is enabling users to scale their update-intensive web services.


The webinar starts at 09:00 Pacific/17:00 UK/18:00 CET today (June 9th 2010).


Still time to register (for free) at http://www.mysql.com/news-and-events/web-seminars/display-545.html – even if you can’t attend, this way you’ll get sent a link to the charts and replay.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Andrew Morgan";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:30;a:6:{s:4:"data";s:88:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:27:"iPhone 4: First Impressions";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:60:"tag:typepad.com,2003:post-6a00d83452e46469e2013484e4499e970c";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:58:"http://www.theopenforce.com/2010/06/hands-on-iphone-4.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:9427:"I managed to get an "early upgrade" of my iPhone 3GS to the iPhone 4 despite AT&T's best efforts.  I've had a couple of days using the new iOS 4 operating system on my 3GS and a couple of hours with the iPhone 4.  So here are a few highlights of the initial hands-on experience with more updates on the weekend.

Updated with additional information, video & photos.

Low-res video (VGA resolution):





Here's a gallery of additional photos from the iPhone 4:(Double click to see larger versions.)



Upgrade Process

Other than AT&T's longstanding inability to deal with demand, the upgrade process is pretty simple.   I chased the UPS driver home to get my iPhone today and just plugged into the USB cable to restore my last iPhone 3GS backup.  That took about 20 minutes.  Applications, data, settings etc were exactly where I left them. However, since I manually manage my MP3 files, it didn't restore those, which is kind of a nuisance.  You also need to re-enter email and WiFi passwords, which makes sense.  During the upgrade process you can sign up for a 60 day trial of MobileMe, which is tempting if you own an iPad and iPhone.  And you also need to re-activate the account by calling a toll-free somewhat-automated AT&T service.  My hold time was just over 3 minutes and then it took another few minutes to go through the terms and conditions.  In fact, I had to type or say my cell number 3 times, zip code twice and agree to the terms twice.  All told it took about 10 minutes. But considering it's AT&T, it could have been worse.

But then a few minutes after syncing, I noticed that not all of my applications were restored. The New York Times, Frotz, Wikipanion, Engadget, Guitar Tab Toolkit and several others apps were missing.  Not quite sure why.  So I plugged in the USB cable a second time, canceled the backup and suddenly the remaining apps were being restored.  That took another 20 minutes.  Not sure if I did something wrong here or the iTunes Store was overloaded.  But if some of your apps aren't restored initially, don't panic.  But if this happens you'll also have to re-arrange the app icons back to how you used to have them.  





Hardware

Better battery life, better screen, better audio, better camera and for those who actually need to talk on their phone, better cellular coverage.  Admittedly, it's still AT&T, but I believe the new antennae built into the casing will help. On my 3GS, I've had calls drop 4 or 5 times while driving on 280 (which has a black-hole for cell service near Sand Hill Road.)  But so far, so good.

The iPhone 4 is slightly skinnier than it's predecessor, and a bit more squared off, but to me the differences are subtle.  Its the same weight and doesn't really feel much thinner, not that that was an issue.  If you had a third-party case for your old iPhone it may or may not fit the new one, depending on how snug it was to begin with.  My old soft rubberized case seems to hang a bit loose, like pants a size too large, but it's not far off. If you're into design then yes, the iPhone 4 has got a modern-retro cool style.  But to me it's not a big deal.

The new screen is better, but again, it's a fairly subtle improvement. However web sites with small fonts, like the mobile version of TechCrunch, are definitely more readable.  And even existing built-in apps benefit from the higher res fonts.  In side by side comparison, the new screen is sharper and seems to have better contrast, making it easier to read.  For news applications, it's almost like reading a printed magazine, albeit a very small, fussy one.

Similarly, performance is a little faster for some apps.  For example, Google maps screen refresh is noticeably snappier than before.  And in side by side comparisons, for example, updating stories from the New York Times or AllThingsD, the iPhone 4 is consistently faster. Not a lot faster, and not in itself enough to make a huge fuss about, but I'll take it.  

The camera, on the other hand, is noticeably improved.  I often end up at conferences panels or blues clubs where I don't always have my trusty Canon G9.  In these cases, the lighting conditions are never ideal and as a result, the iPhone 3GS camera just doesn't cut it.  And in my experience, the 3GS video was completely useless as any volume of live music (say 100 db or louder) gets clipped and distorted by the built-in microphone.  

The iPhone 4's camera is much improved. The pictures are high-res (5 mb versus 3) but the real improvement comes from being more sensitive in low light conditions.  The iPhone 4 also has a front VGA (640x480) low-res camera used by the FaceTime video conference call app that is also suitable for quick self-portraits with less fumbling around.  While the camera isn't perfect, it's miles better than the 3GS and can match low-end point and shoot or Flip video cameras. I've posted two photos in crappy lighting and the iPhone 4 makes a decent job of it.

We'll see how the video does in a live music situation on the weekend.

Sample high-res shot:


   

Sample low-res VGA shot:


 

And here's a quick and dirty use of the low-res VGA video capability:(My apologies for the guitar playing!)





Improved Cellular Reception

While there has been some questions and comments about reduced cell reception depending on how you hold the iPhone, I haven't had any problems.  (Hint: avoid directly touching the antenna in the lower left corner when you hold the phone.)

In fact, as the video at the top of the post demonstrates, I was able to make continuous calls on several notorious silicon valley dead spots, including Highway 280 near Sand Hill Rd and Highway 17 to Santa Cruz.  However, I did lose reception in a tunnel (to be expected) and on rural Highway 9 in Saratoga. But even that was just one disconnect compared to the normal 3 drops I used to experience.  

If you've been frustrated by dropped calls with the 3GS, this improvement alone may be worth the upgrade price.

The speaker is also slightly clearer which is useful if you do conference calls or play music from the speaker (which I do on occasion.)









Software

The new iOS 4 is good on the 3GS but it really rocks on the new hardware.  Not only is the multi-tasking quick, but the better hardware makes even existing applications look better and run faster.  Hopefully in the weeks to follow we'll see more applications updated to use the new multi-tasking.  

Note that the multi-tasking on the iPhone is not the same kind of flat-out full-on multi-tasking you may be used to on a desktop compute.  It's really more of an intelligent quick-restore of an application with some limited multi-tasking for maintaining cell connection, playing music, getting notifications etc.  On a handset, this seems to work fine.  It's not like I need a massive spreadsheet to recalc or some kind of long-running DBMS transactions to go run in separate threads.  But we'll see in the fall whether this same model works as well on the iPad.

Nonetheless, the multi-tasking, is a intuitive as you could imagine.  Double click the iPhone button below the screen to pull up your recent or running applications.  So you don't have to go back through the home screen and scroll through pages of apps when you, say, confirm a calendar appointment in an email while talking on the phone and looking at a map.  It's not as good as multiple desktop apps on the screen at once, but the experience works well on the small screen of a smartphone.ns.

Some of the built-in apps are also improved.  For example, iPhone email now has a unified in-box and threaded conversations. With a unified in-box, I can now finally start to move off hotmail and over to gmail without having to manually check email in two places.

You can also now run the iBooks application on the iPhone with bookmarks and content synchronized.

If you're happy with your 3GS and just want multi-tasking and a unified email inbox with threaded messages, and iBooks, you can get all of that with an upgrade to the iOS 4 platform for free.

I'll take a look at some of the new apps like FaceTime on the weekend.









Bottom Line

Overall, iPhone 4 is an incremental improvement.  I am not sure whether I would label it game changing; that depends on how much you use FaceTime, iMovie or other new applications that have yet to be created.  But it is certainly a worthwhile upgrade, just to get the improved battery life, camera and improved cellular reception.  But if you do the upgrade, note that it can take about an hour to backup your old phone, restore on the new one and activate the account with AT&T.  Don't attempt this if you need to use the new phone in 10 minutes.

If you can get the subsidized price or early upgrade and can live with AT&T, then it's $200 well spent. Otherwise, you may have to wait for Verizon to pick it up next year.  

With the iPhone 4, Apple has once again set and raised the bar.



Note: I'll update this review again on the weekend.






Reviews: Engadget, Gizmodo Review Roundup, NY Times
Mercury News: Apple sells 3 million iPads, Verizon iPhone Next Year?



";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Sat, 26 Jun 2010 01:33:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:10:{i:0;a:5:{s:4:"data";s:13:"Hot Companies";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:4:"Misc";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:10:"Technology";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:14:"apple iphone 4";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:8:"AT&T";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:18:"cellular reception";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:8:"hands on";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:21:"highway 17 santa cruz";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:5:"ios 4";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:9;a:5:{s:4:"data";s:14:"sand hill road";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:11631:"
I managed to get an "early upgrade" of my iPhone 3GS to the iPhone 4 despite AT&T's best efforts.  I've had a couple of days using the new iOS 4 operating system on my 3GS and a couple of hours with the iPhone 4.  So here are a few highlights of the initial hands-on experience with more updates on the weekend.



Updated with additional information, video & photos.



Low-res video (VGA resolution):









Here's a gallery of additional photos from the iPhone 4:
(Double click to see larger versions.)








Upgrade Process



Other than AT&T's longstanding inability to deal with demand, the upgrade process is pretty simple.   I chased the UPS driver home to get my iPhone today and just plugged into the USB cable to restore my last iPhone 3GS backup.  That took about 20 minutes.  Applications, data, settings etc were exactly where I left them. However, since I manually manage my MP3 files, it didn't restore those, which is kind of a nuisance.  You also need to re-enter email and WiFi passwords, which makes sense.  During the upgrade process you can sign up for a 60 day trial of MobileMe, which is tempting if you own an iPad and iPhone.  And you also need to re-activate the account by calling a toll-free somewhat-automated AT&T service.  My hold time was just over 3 minutes and then it took another few minutes to go through the terms and conditions.  In fact, I had to type or say my cell number 3 times, zip code twice and agree to the terms twice.  All told it took about 10 minutes. But considering it's AT&T, it could have been worse.



But then a few minutes after syncing, I noticed that not all of my applications were restored. The New York Times, Frotz, Wikipanion, Engadget, Guitar Tab Toolkit and several others apps were missing.  Not quite sure why.  So I plugged in the USB cable a second time, canceled the backup and suddenly the remaining apps were being restored.  That took another 20 minutes.  Not sure if I did something wrong here or the iTunes Store was overloaded.  But if some of your apps aren't restored initially, don't panic.  But if this happens you'll also have to re-arrange the app icons back to how you used to have them.  









Hardware



Better battery life, better screen, better audio, better camera and for those who actually need to talk on their phone, better cellular coverage.  Admittedly, it's still AT&T, but I believe the new antennae built into the casing will help. On my 3GS, I've had calls drop 4 or 5 times while driving on 280 (which has a black-hole for cell service near Sand Hill Road.)  But so far, so good.



The iPhone 4 is slightly skinnier than it's predecessor, and a bit more squared off, but to me the differences are subtle.  Its the same weight and doesn't really feel much thinner, not that that was an issue.  If you had a third-party case for your old iPhone it may or may not fit the new one, depending on how snug it was to begin with.  My old soft rubberized case seems to hang a bit loose, like pants a size too large, but it's not far off. If you're into design then yes, the iPhone 4 has got a modern-retro cool style.  But to me it's not a big deal.



The new screen is better, but again, it's a fairly subtle improvement. However web sites with small fonts, like the mobile version of TechCrunch, are definitely more readable.  And even existing built-in apps benefit from the higher res fonts.  In side by side comparison, the new screen is sharper and seems to have better contrast, making it easier to read.  For news applications, it's almost like reading a printed magazine, albeit a very small, fussy one.



Similarly, performance is a little faster for some apps.  For example, Google maps screen refresh is noticeably snappier than before.  And in side by side comparisons, for example, updating stories from the New York Times or AllThingsD, the iPhone 4 is consistently faster. Not a lot faster, and not in itself enough to make a huge fuss about, but I'll take it.  



The camera, on the other hand, is noticeably improved.  I often end up at conferences panels or blues clubs where I don't always have my trusty Canon G9.  In these cases, the lighting conditions are never ideal and as a result, the iPhone 3GS camera just doesn't cut it.  And in my experience, the 3GS video was completely useless as any volume of live music (say 100 db or louder) gets clipped and distorted by the built-in microphone.  



The iPhone 4's camera is much improved. The pictures are high-res (5 mb versus 3) but the real improvement comes from being more sensitive in low light conditions.  The iPhone 4 also has a front VGA (640x480) low-res camera used by the FaceTime video conference call app that is also suitable for quick self-portraits with less fumbling around.  While the camera isn't perfect, it's miles better than the 3GS and can match low-end point and shoot or Flip video cameras. I've posted two photos in crappy lighting and the iPhone 4 makes a decent job of it.



We'll see how the video does in a live music situation on the weekend.



Sample high-res shot:




Zack_back_camera 
 
 



Sample low-res VGA shot:




Zack_front_camera 



And here's a quick and dirty use of the low-res VGA video capability:
(My apologies for the guitar playing!)









Improved Cellular Reception



While there has been some questions and comments about reduced cell reception depending on how you hold the iPhone, I haven't had any problems.  (Hint: avoid directly touching the antenna in the lower left corner when you hold the phone.)




In fact, as the video at the top of the post demonstrates, I was able to make continuous calls on several notorious silicon valley dead spots, including Highway 280 near Sand Hill Rd and Highway 17 to Santa Cruz.  However, I did lose reception in a tunnel (to be expected) and on rural Highway 9 in Saratoga. But even that was just one disconnect compared to the normal 3 drops I used to experience.  



If you've been frustrated by dropped calls with the 3GS, this improvement alone may be worth the upgrade price.



The speaker is also slightly clearer which is useful if you do conference calls or play music from the speaker (which I do on occasion.)
















Software



The new iOS 4 is good on the 3GS but it really rocks on the new hardware.  Not only is the multi-tasking quick, but the better hardware makes even existing applications look better and run faster.  Hopefully in the weeks to follow we'll see more applications updated to use the new multi-tasking.  



Note that the multi-tasking on the iPhone is not the same kind of flat-out full-on multi-tasking you may be used to on a desktop compute.  It's really more of an intelligent quick-restore of an application with some limited multi-tasking for maintaining cell connection, playing music, getting notifications etc.  On a handset, this seems to work fine.  It's not like I need a massive spreadsheet to recalc or some kind of long-running DBMS transactions to go run in separate threads.  But we'll see in the fall whether this same model works as well on the iPad.



Nonetheless, the multi-tasking, is a intuitive as you could imagine.  Double click the iPhone button below the screen to pull up your recent or running applications.  So you don't have to go back through the home screen and scroll through pages of apps when you, say, confirm a calendar appointment in an email while talking on the phone and looking at a map.  It's not as good as multiple desktop apps on the screen at once, but the experience works well on the small screen of a smartphone.ns.



Some of the built-in apps are also improved.  For example, iPhone email now has a unified in-box and threaded conversations. With a unified in-box, I can now finally start to move off hotmail and over to gmail without having to manually check email in two places.



You can also now run the iBooks application on the iPhone with bookmarks and content synchronized.



If you're happy with your 3GS and just want multi-tasking and a unified email inbox with threaded messages, and iBooks, you can get all of that with an upgrade to the iOS 4 platform for free.



I'll take a look at some of the new apps like FaceTime on the weekend.
















Bottom Line



Overall, iPhone 4 is an incremental improvement.  I am not sure whether I would label it game changing; that depends on how much you use FaceTime, iMovie or other new applications that have yet to be created.  But it is certainly a worthwhile upgrade, just to get the improved battery life, camera and improved cellular reception.  But if you do the upgrade, note that it can take about an hour to backup your old phone, restore on the new one and activate the account with AT&T.  Don't attempt this if you need to use the new phone in 10 minutes.



If you can get the subsidized price or early upgrade and can live with AT&T, then it's $200 well spent. Otherwise, you may have to wait for Verizon to pick it up next year.  



With the iPhone 4, Apple has once again set and raised the bar.







Note: I'll update this review again on the weekend.














PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Zack Urlocker";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:31;a:6:{s:4:"data";s:33:"
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:2:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:44:"Using PMP to double MySQL throughput, part 2";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:53:"http://www.facebook.com/note.php?note_id=405092575932";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:53:"http://www.facebook.com/note.php?note_id=405092575932";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:7894:"With a few changes to InnoDB and MySQL I think I can get more than 100,000 disk reads/second from MySQL 5.1.47. InnoDB and MySQL continue to amaze me. This is a remarkable result.

Storage devices that support more than 100,000 IOPs can be purchased for commodity servers. This capacity is much more interesting when InnoDB can use all of it or when storage vendors give you a discount because your software can't use their capacity. My first tests were done with an 8-core server and I was able to get ~12,000 disk reads/second from InnoDB in MySQL 5.0.84 and ~18,000 reads/second from the InnoDB plugin in MySQL 5.1.47. My second tests were done on a 16-core server and I was able to get ~18,000 reads/second from 5.0.84, ~30,000 reads/second from 5.1.47 and ~70,000 reads/second from 5.5.4m3. All of these results were measured using super-fast storage -- a 40G tmpfs partition on server with 72G RAM.

I then used PMP to debug mysqld performance during the tests and found a few interesting problems. The clients used sysbench as described here with a few changes. First, the test table had 128M rows and used ~30G in InnoDB. The InnoDB buffer pool was 4G so that most queries required IO between tmpfs and the InnoDB buffer pool.

The first thing that I noticed is that 5.1.47 with the Facebook patch used much less CPU to produce the same throughput when using a flash device. Before I made other changes the primary reason for this is the innodb_fast_checksum patch. This reduces the CPU cost of computing checksums by 10X. This is a good start as a server with more spare CPU capacity should be able to handle more concurrency before saturating.

Then I began to look at PMP stacks and oprofile output and noticed a few problems. The first problem is that a lot of time was spent in ut_2_power_up. This is called by BUF_READ_AHEAD_AREA which is a macro that hides a function call. Unless  buf_pool->curr_size is small, the value of BUF_READ_AHEAD_AREA is always 64. As far as I know, buf_pool->curr_size is the number of pages in the buffer pool. It must be less than 1024 for the value of BUF_READ_AHEAD_AREA to be less than 64. I think it is better to change this to always be 64 and the change increased peak QPS at 64 concurrent threads from 38,520 to 51,593. I opened feature request 54814 for this.

/** The size in pages of the area which the read-ahead algorithms read if
invoked */
#define BUF_READ_AHEAD_AREA                                     \
        ut_min(64, ut_2_power_up(buf_pool->curr_size / 32))

The next problem that I saw was too much time spent in buf_flush_free_margin. This function checks the end of the buffer pool LRU for dirty blocks and issues write requests for them. Blocks at the end of the LRU are the first to be moved to the free list. When a thread issues one or more read requests is then calls buf_flush_free_margin to make sure there are entries on the free list or non-dirty pages at the end of the LRU . The function buf_flush_LRU_recommendation is called by buf_flush_free_margin to count the number of pages on the free list and the end of the LRU that are ready to be replaced. It holds the buffer pool mutex while making this check. I changed it to read the length of the free list before locking the buffer pool mutex and return immediately when there are more than enough pages on the free list. This reduces mutex contention on a highly-contended mutex and saves more CPU. The change is listed below. I think it is safe, but it needs to be reviewed. Peak QPS did not increase with this change, but it reduced CPU utilization by ~10%. I opened feature request 54815 for this.

        if (UT_LIST_GET_LEN(buf_pool->free) >= n_needed) {

                /* This does a dirty read of buf_pool->free. That is good
                enough and reduces mutex contention on buf_pool->mutex. */

                return 0;
        }

I checked the functions that call buf_flush_free_margin. It does a lot of work when there are fewer than 37 pages ready to be replaced between the free list and the end of the LRU. However, most of the callers were reading one page. The code for readahead is the exception. I changed buf_flush_free_margin to take the number of pages needed as an argument and only when that many pages were not ready to be replaced would it go to the step where it does a lot of work. As most callers need but one page, this increased peak QPS at 64 threads to 74,309.

Finally I applied the change to reduce fcntl calls as described here and peak QPS at 64 threads increased to 118,858. This is a remarkable result as disk reads/second are about 90% of QPS for this benchmark. 

Full results are listed below as QPS from sysbench. Each query fetches one row by primary key using HANDLER. The database was ~30G with ~29G for the PK index and the InnoDB buffer pool was 4G. QPS is listed for 1, 2, 4, 8, 16, 32, 64, 128 and 256 threads. Unmodified 5.1.47 with the InnoDB plugin was tested and is listed as 5147orig in the results below. Five patched binaries were also tested. Each binary includes all of the patches from its predecessor and then adds one change. The binaries are:

5147fb.base - 5.1.47 with the Facebook patch
5147fb.bufra - 5147fb.base + change to BUF_READ_AHEAD_AREA
5147fb.skip - 5147fb.bufra + shortcut in buf_flush_LRU_recommendation
5147fb.npages - 5147fb.skip + buf_flush_free_margin takes needs_pages argument
5147fb.fcntl - 5147fb.npages + change to reduce calls to fcntl

In addition to higher peak QPS and lower CPU utilization, there is much less contention on the buffer pool mutex. From SHOW MUTEX STATUS there are ~53m waits on it using 5147fb.base and ~130k using 5147fb.fcntl.

The QPS results with a graph of the results here.

876     1843    3818    10395   22713   30424   34524   34771   34827   5147orig
919     1974    3887    9900    22792   34944   38520   38777   36876   5147fb.base
974     2061    3886    9680    23488   48903   51953   51419   48423   5147fb.bufra
941     2028    4079    9803    22411   48795   51958   51592   48534   5147fb.skip
943     1980    3911    9731    24528   74389   74309   70266   67240   5147fb.npages
864     1999    4013    9281    21366   73939   118858  113052  87093   5147fb.fcntl

This lists 95th percentile response time in microseconds as reported by sysbench:

2070    2000    1970    1210    1120    1920    3700    7990    16560   5147orig
2040    1920    1930    1250    1130    1740    3590    7890    17410   5147fb.base
2020    1900    1930    1220    1110    1140    2690    6180    13740   5147fb.bufra
2020    1840    1890    1200    1130    1140    2690    6150    13760   5147fb.skip
2020    1940    1930    1180    1080    780     1610    3680    8050    5147fb.npages
2110    1930    1910    1310    1150    800     1010    3680    9940    5147fb.fcntl

The sysbench command line for the test. Use this version of sysbench.

# Run for nt in 1, 2, 4, 8, 16, 32, 64, 128, 256
sysbench3 --test=oltp --oltp-table-size=128000000 --max-time=180 --max-requests=0 \
    --mysql-table-engine=innodb --db-ps-mode=disable --mysql-engine-trx=yes \
    --oltp-read-only --oltp-skip-trx --oltp-test-mode=simple \
    --oltp-point-select-mysql-handler --oltp-dist-type=uniform --oltp-range-size=1 \
    --num-threads=$nt --seed-rng=1 run

The configure command line:

b="/data/$1"; ./configure --without-debug --enable-thread-safe-client\
     --prefix=$b --exec-prefix=$b \
    --with-plugins=partition,csv,blackhole,myisam,heap,innodb_plugin,pbxt \
    --without-plugin-innobase --with-unix-socket-path=$b/var/mysql.sock \
    --with-fast-mutexes --with-extra-charsets=all \
     C_EXTRA_FLAGS="-g -O2 -fno-omit-frame-pointer -fno-strict-aliasing -DNO_ALARM -DSIGNAL_WITH_VIO_CLOSE -Wall"

The my.cnf settings:

innodb_buffer_pool_size=4000M
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
max_connections=2000
table_cache=2000
";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 17:35:13 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:9594:"With a few changes to InnoDB and MySQL I think I can get more than 100,000 disk reads/second from MySQL 5.1.47. InnoDB and MySQL continue to amaze me. This is a remarkable result.

Storage devices that support more than 100,000 IOPs can be purchased for commodity servers. This capacity is much more interesting when InnoDB can use all of it or when storage vendors give you a discount because your software can't use their capacity. My first tests were done with an 8-core server and I was able to get ~12,000 disk reads/second from InnoDB in MySQL 5.0.84 and ~18,000 reads/second from the InnoDB plugin in MySQL 5.1.47. My second tests were done on a 16-core server and I was able to get ~18,000 reads/second from 5.0.84, ~30,000 reads/second from 5.1.47 and ~70,000 reads/second from 5.5.4m3. All of these results were measured using super-fast storage -- a 40G tmpfs partition on server with 72G RAM.

I then used PMP to debug mysqld performance during the tests and found a few interesting problems. The clients used sysbench as described here with a few changes. First, the test table had 128M rows and used ~30G in InnoDB. The InnoDB buffer pool was 4G so that most queries required IO between tmpfs and the InnoDB buffer pool.

The first thing that I noticed is that 5.1.47 with the Facebook patch used much less CPU to produce the same throughput when using a flash device. Before I made other changes the primary reason for this is the innodb_fast_checksum patch. This reduces the CPU cost of computing checksums by 10X. This is a good start as a server with more spare CPU capacity should be able to handle more concurrency before saturating.

Then I began to look at PMP stacks and oprofile output and noticed a few problems. The first problem is that a lot of time was spent in ut_2_power_up. This is called by BUF_READ_AHEAD_AREA which is a macro that hides a function call. Unless  buf_pool->curr_size is small, the value of BUF_READ_AHEAD_AREA is always 64. As far as I know, buf_pool->curr_size is the number of pages in the buffer pool. It must be less than 1024 for the value of BUF_READ_AHEAD_AREA to be less than 64. I think it is better to change this to always be 64 and the change increased peak QPS at 64 concurrent threads from 38,520 to 51,593. I opened feature request 54814 for this.

/** The size in pages of the area which the read-ahead algorithms read if
invoked */
#define BUF_READ_AHEAD_AREA                                     \
        ut_min(64, ut_2_power_up(buf_pool->curr_size / 32))


The next problem that I saw was too much time spent in buf_flush_free_margin. This function checks the end of the buffer pool LRU for dirty blocks and issues write requests for them. Blocks at the end of the LRU are the first to be moved to the free list. When a thread issues one or more read requests is then calls buf_flush_free_margin to make sure there are entries on the free list or non-dirty pages at the end of the LRU . The function buf_flush_LRU_recommendation is called by buf_flush_free_margin to count the number of pages on the free list and the end of the LRU that are ready to be replaced. It holds the buffer pool mutex while making this check. I changed it to read the length of the free list before locking the buffer pool mutex and return immediately when there are more than enough pages on the free list. This reduces mutex contention on a highly-contended mutex and saves more CPU. The change is listed below. I think it is safe, but it needs to be reviewed. Peak QPS did not increase with this change, but it reduced CPU utilization by ~10%. I opened feature request 54815 for this.

        if (UT_LIST_GET_LEN(buf_pool->free) >= n_needed) {

                /* This does a dirty read of buf_pool->free. That is good
                enough and reduces mutex contention on buf_pool->mutex. */

                return 0;
        }


I checked the functions that call buf_flush_free_margin. It does a lot of work when there are fewer than 37 pages ready to be replaced between the free list and the end of the LRU. However, most of the callers were reading one page. The code for readahead is the exception. I changed buf_flush_free_margin to take the number of pages needed as an argument and only when that many pages were not ready to be replaced would it go to the step where it does a lot of work. As most callers need but one page, this increased peak QPS at 64 threads to 74,309.

Finally I applied the change to reduce fcntl calls as described here and peak QPS at 64 threads increased to 118,858. This is a remarkable result as disk reads/second are about 90% of QPS for this benchmark. 

Full results are listed below as QPS from sysbench. Each query fetches one row by primary key using HANDLER. The database was ~30G with ~29G for the PK index and the InnoDB buffer pool was 4G. QPS is listed for 1, 2, 4, 8, 16, 32, 64, 128 and 256 threads. Unmodified 5.1.47 with the InnoDB plugin was tested and is listed as 5147orig in the results below. Five patched binaries were also tested. Each binary includes all of the patches from its predecessor and then adds one change. The binaries are:

In addition to higher peak QPS and lower CPU utilization, there is much less contention on the buffer pool mutex. From SHOW MUTEX STATUS there are ~53m waits on it using 5147fb.base and ~130k using 5147fb.fcntl.

The QPS results with a graph of the results here.

876     1843    3818    10395   22713   30424   34524   34771   34827   5147orig
919     1974    3887    9900    22792   34944   38520   38777   36876   5147fb.base
974     2061    3886    9680    23488   48903   51953   51419   48423   5147fb.bufra
941     2028    4079    9803    22411   48795   51958   51592   48534   5147fb.skip
943     1980    3911    9731    24528   74389   74309   70266   67240   5147fb.npages
864     1999    4013    9281    21366   73939   118858  113052  87093   5147fb.fcntl


This lists 95th percentile response time in microseconds as reported by sysbench:

2070    2000    1970    1210    1120    1920    3700    7990    16560   5147orig
2040    1920    1930    1250    1130    1740    3590    7890    17410   5147fb.base
2020    1900    1930    1220    1110    1140    2690    6180    13740   5147fb.bufra
2020    1840    1890    1200    1130    1140    2690    6150    13760   5147fb.skip
2020    1940    1930    1180    1080    780     1610    3680    8050    5147fb.npages
2110    1930    1910    1310    1150    800     1010    3680    9940    5147fb.fcntl


The sysbench command line for the test. Use this version of sysbench.

# Run for nt in 1, 2, 4, 8, 16, 32, 64, 128, 256
sysbench3 --test=oltp --oltp-table-size=128000000 --max-time=180 --max-requests=0 \
    --mysql-table-engine=innodb --db-ps-mode=disable --mysql-engine-trx=yes \
    --oltp-read-only --oltp-skip-trx --oltp-test-mode=simple \
    --oltp-point-select-mysql-handler --oltp-dist-type=uniform --oltp-range-size=1 \
    --num-threads=$nt --seed-rng=1 run


The configure command line:

b="/data/$1"; ./configure --without-debug --enable-thread-safe-client\
     --prefix=$b --exec-prefix=$b \
    --with-plugins=partition,csv,blackhole,myisam,heap,innodb_plugin,pbxt \
    --without-plugin-innobase --with-unix-socket-path=$b/var/mysql.sock \
    --with-fast-mutexes --with-extra-charsets=all \
     C_EXTRA_FLAGS="-g -O2 -fno-omit-frame-pointer -fno-strict-aliasing -DNO_ALARM -DSIGNAL_WITH_VIO_CLOSE -Wall"


The my.cnf settings:

innodb_buffer_pool_size=4000M
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
max_connections=2000
table_cache=2000


PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:32;a:6:{s:4:"data";s:98:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:17:"MySQL and Quoting";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:36:"http://www.pythian.com/news/?p=13811";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:52:"http://www.pythian.com/news/13811/mysql-and-quoting/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:4657:"MySQL does not follow the ANSI SQL standard for quoting.  MySQL’s default quoting behavior is that either single or double quotes can be used to quote a string (this gets me into trouble when I work with Oracle databases, as double quotes do not indicate a string!).  
mysql> SELECT 'alive';
+-------+
| alive |
+-------+
| alive |
+-------+
1 row in set (0.00 sec)

mysql> SELECT "alive";
+-------+
| alive |
+-------+
| alive |
+-------+
1 row in set (0.00 sec)
Bare words are dealt with in context; in this case, a bare word would be parsed as a column name:
mysql> SELECT alive;
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'
Backquotes are the way MySQL escapes table names.  So, if you want a reserved word, number or operator to be the name of an object (ie, a table named “1″ or a column named “date”) you need to use backquotes to avoid a syntax error….for example:

mysql> SELECT `alive`;
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'

mysql> CREATE TABLE table (column date);
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'table (column date)' at line 1

mysql> CREATE TABLE `table` (column date);
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'column date)' at line 1

mysql> CREATE TABLE `table` (`column` date);
Query OK, 0 rows affected (0.11 sec)

mysql> DROP TABLE table;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'table' at line 1

mysql> DROP TABLE `table`;
Query OK, 0 rows affected (0.00 sec)
The above illustrates MySQL’s default behavior, which in summary is:
- With no quotes or backquotes, MySQL treats the word ‘alive’ as a field name.
- With single or double quotes, MySQL treats the word ‘alive’ as a string.
Changing SQL Mode
However, this behavior can be changed by setting the sql_mode.  There are over 30 different sql modes in MySQL 5.1 that change the behavior of MySQL (not all are related to quoting).  The sql_mode value is a comma-separated list showing which sql modes are used.
The default sql_mode is blank, allowing default MySQL behavior.
If sql_mode is changed to ANSI_QUOTES, then ANSI quotes are used….
mysql> SET SESSION SQL_MODE='ANSI_QUOTES';
Query OK, 0 rows affected (0.00 sec)

mysql> SHOW SESSION VARIABLES LIKE 'sql_mode';
+---------------+-------------+
| Variable_name | Value       |
+---------------+-------------+
| sql_mode      | ANSI_QUOTES |
+---------------+-------------+
1 row in set (0.00 sec)
MySQL still treats barewords and backquotes as columns:
mysql> SELECT alive;
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'
mysql> SELECT `alive`;
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'
MySQL still treats single quotes as strings:
mysql> SELECT 'alive';
+-------+
| alive |
+-------+
| alive |
+-------+
1 row in set (0.00 sec)
However, now MySQL treats double quotes as escape characters, ie, treating double quotes like backquotes:
mysql> SELECT "alive";
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'
Putting quote marks in strings
Since I am discussing quoting, I will end with how to put quote marks into strings.
There are 3 ways to embed quotes in a string:
1) mix single and double quotes:
mysql> SELECT 'I say "OK".' , "I say 'OK'." , 'I say ''OK''.';
+-------------+-------------+-------------+
| I say "OK". | I say 'OK'. | I say 'OK'. |
+-------------+-------------+-------------+
| I say "OK". | I say 'OK'. | I say 'OK'. |
+-------------+-------------+-------------+
1 row in set (0.00 sec)
2) Use 2 quotes in a row to escape quotes:
mysql> SELECT 'I say ''OK''.' , "I say ""OK"".";
+-------------+-------------+
| I say 'OK'. | I say "OK". |
+-------------+-------------+
| I say 'OK'. | I say "OK". |
+-------------+-------------+
1 row in set (0.00 sec)
3) Use the backslash to escape quotes:
mysql> SELECT 'I say \'OK\'.' , "I say \"OK\".";
+-------------+-------------+
| I say 'OK'. | I say "OK". |
+-------------+-------------+
| I say 'OK'. | I say "OK". |
+-------------+-------------+
1 row in set (0.00 sec)
(Interestingly enough, I wrote a blog post on this topic almost exactly 2 years ago, too!)";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 16:43:15 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:12:{i:0;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:6:"Oracle";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:14:"Technical Blog";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:4:"ANSI";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:13:"ANSI standard";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:11:"ansi_quotes";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:6:"escape";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:5:"quote";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:7:"quoting";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:9;a:5:{s:4:"data";s:12:"SQL standard";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:10;a:5:{s:4:"data";s:8:"SQL_MODE";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:11;a:5:{s:4:"data";s:8:"standard";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:5239:"
MySQL does not follow the ANSI SQL standard for quoting.  MySQL’s default quoting behavior is that either single or double quotes can be used to quote a string (this gets me into trouble when I work with Oracle databases, as double quotes do not indicate a string!).  


mysql> SELECT 'alive';
+-------+
| alive |
+-------+
| alive |
+-------+
1 row in set (0.00 sec)

mysql> SELECT "alive";
+-------+
| alive |
+-------+
| alive |
+-------+
1 row in set (0.00 sec)


Bare words are dealt with in context; in this case, a bare word would be parsed as a column name:


mysql> SELECT alive;
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'


Backquotes are the way MySQL escapes table names.  So, if you want a reserved word, number or operator to be the name of an object (ie, a table named “1″ or a column named “date”) you need to use backquotes to avoid a syntax error….for example:




mysql> SELECT `alive`;
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'

mysql> CREATE TABLE table (column date);
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'table (column date)' at line 1

mysql> CREATE TABLE `table` (column date);
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'column date)' at line 1

mysql> CREATE TABLE `table` (`column` date);
Query OK, 0 rows affected (0.11 sec)

mysql> DROP TABLE table;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'table' at line 1

mysql> DROP TABLE `table`;
Query OK, 0 rows affected (0.00 sec)


The above illustrates MySQL’s default behavior, which in summary is:

- With no quotes or backquotes, MySQL treats the word ‘alive’ as a field name.

- With single or double quotes, MySQL treats the word ‘alive’ as a string.


Changing SQL Mode

However, this behavior can be changed by setting the sql_mode.  There are over 30 different sql modes in MySQL 5.1 that change the behavior of MySQL (not all are related to quoting).  The sql_mode value is a comma-separated list showing which sql modes are used.


The default sql_mode is blank, allowing default MySQL behavior.


If sql_mode is changed to ANSI_QUOTES, then ANSI quotes are used….


mysql> SET SESSION SQL_MODE='ANSI_QUOTES';
Query OK, 0 rows affected (0.00 sec)

mysql> SHOW SESSION VARIABLES LIKE 'sql_mode';
+---------------+-------------+
| Variable_name | Value       |
+---------------+-------------+
| sql_mode      | ANSI_QUOTES |
+---------------+-------------+
1 row in set (0.00 sec)


MySQL still treats barewords and backquotes as columns:


mysql> SELECT alive;
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'
mysql> SELECT `alive`;
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'


MySQL still treats single quotes as strings:


mysql> SELECT 'alive';
+-------+
| alive |
+-------+
| alive |
+-------+
1 row in set (0.00 sec)


However, now MySQL treats double quotes as escape characters, ie, treating double quotes like backquotes:


mysql> SELECT "alive";
ERROR 1054 (42S22): Unknown column 'alive' in 'field list'


Putting quote marks in strings


Since I am discussing quoting, I will end with how to put quote marks into strings.


There are 3 ways to embed quotes in a string:


1) mix single and double quotes:


mysql> SELECT 'I say "OK".' , "I say 'OK'." , 'I say ''OK''.';
+-------------+-------------+-------------+
| I say "OK". | I say 'OK'. | I say 'OK'. |
+-------------+-------------+-------------+
| I say "OK". | I say 'OK'. | I say 'OK'. |
+-------------+-------------+-------------+
1 row in set (0.00 sec)


2) Use 2 quotes in a row to escape quotes:


mysql> SELECT 'I say ''OK''.' , "I say ""OK"".";
+-------------+-------------+
| I say 'OK'. | I say "OK". |
+-------------+-------------+
| I say 'OK'. | I say "OK". |
+-------------+-------------+
1 row in set (0.00 sec)


3) Use the backslash to escape quotes:


mysql> SELECT 'I say \'OK\'.' , "I say \"OK\".";
+-------------+-------------+
| I say 'OK'. | I say "OK". |
+-------------+-------------+
| I say 'OK'. | I say "OK". |
+-------------+-------------+
1 row in set (0.00 sec)


(Interestingly enough, I wrote a blog post on this topic almost exactly 2 years ago, too!)

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:16:"Sheeri K. Cabral";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:33;a:6:{s:4:"data";s:78:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:52:"Log Buffer #192, A Carnival of The Vanities for DBAs";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:36:"http://www.pythian.com/news/?p=13285";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:85:"http://www.pythian.com/news/13285/log-buffer-192-a-carnival-of-the-vanities-for-dbas/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:2604:"It’s Friday, and summer’s here.  While it seems the industry is slowing down to a lazy pace, there is still some action so let’s splash right in to this week’s edition of Log Buffer DBA industry news in Log Buffer #192.
Alex Gorbachev had a few minutes to suggest the following interesting tidbits to me before running off to attend Oracle ACE Director activities at ODTUG/Kaleidoscope this weekend. One of these days we’ll have to see if he can share some of what goes on behind closed doors at those hush hush sessions.

He’s happy fellow OakTable member and Oracle ACE Director Tanel Poder is back to blogging with low-level undocumented internals for Oracle troubleshooting geeks.

Also an OakTable Network member, Jonathan Lewis, posted about the experience of one client migrating to 64 bit and increasing their SGA to see performance going down. The moral of this story is make sure your memory configuration is right.
Tom Kyte continues a series of blog posts about things he’s learning and likes, this time on SQL*Plus. And, at the same time includes something he doesn’t like, a feature in Windows 7 that he has mixed feelings on.
In the SQL Server world:
On his blog, In Recovery, Paul S. Randal runs some experiments to explore whether multiple data files can lead to an improvement in performance.
Adam Machanic smashes a DMV Myth: session_id > 50 == User Process on SQL blog.com.
Pinal Dave has two interesting posts this week on SQL Authority.com.  In the first he shares his experience meeting and learning from trainer and industry guru Bryan Oliver. Secondly, he provides valuable advice for SQL Server developers dealing with XML from friend Jacob Sebastian on Select * FROM XML which he suggests looking at before reading his own advice on the same topic.
We continue to see the traction of MySQL within Oracle communities.  More and more user group and Oracle-focused events are adding MySQL content to their agendas.  At ODTUG/Kaleidoscope, Sheeri Cabral and Ronald Bradford were instrumental in developing the MySQL track of 19 sessions, and are speaking themselves.
Aleksander Kuzminsky announced that Percona Server 5.1.47-rel11 is available for download, noting new features and fixed bugs.
Craig Mullins writes how to access your DB2 Catalog “Poster” online for any of you constantly looking for DB2 Catalog table and column names.
Dave Page does a bit of comparison and gives his opinion on VoltDB, the new database server architected by the ‘father’ of Postgres, Dr Michael Stonebraker.
Until next week.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 15:17:32 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:8:{i:0;a:5:{s:4:"data";s:10:"Log Buffer";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:5:"NoSQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:6:"Oracle";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:10:"PostgreSQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:7:"Pythian";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:10:"SQL Server";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:14:"Technical Blog";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:5172:"
It’s Friday, and summer’s here.  While it seems the industry is slowing down to a lazy pace, there is still some action so let’s splash right in to this week’s edition of Log Buffer DBA industry news in Log Buffer #192.


Alex Gorbachev had a few minutes to suggest the following interesting tidbits to me before running off to attend Oracle ACE Director activities at ODTUG/Kaleidoscope this weekend. One of these days we’ll have to see if he can share some of what goes on behind closed doors at those hush hush sessions.



He’s happy fellow OakTable member and Oracle ACE Director Tanel Poder is back to blogging with low-level undocumented internals for Oracle troubleshooting geeks.



Also an OakTable Network member, Jonathan Lewis, posted about the experience of one client migrating to 64 bit and increasing their SGA to see performance going down. The moral of this story is make sure your memory configuration is right.


Tom Kyte continues a series of blog posts about things he’s learning and likes, this time on SQL*Plus. And, at the same time includes something he doesn’t like, a feature in Windows 7 that he has mixed feelings on.


In the SQL Server world:


On his blog, In Recovery, Paul S. Randal runs some experiments to explore whether multiple data files can lead to an improvement in performance.


Adam Machanic smashes a DMV Myth: session_id > 50 == User Process on SQL blog.com.


Pinal Dave has two interesting posts this week on SQL Authority.com.  In the first he shares his experience meeting and learning from trainer and industry guru Bryan Oliver. Secondly, he provides valuable advice for SQL Server developers dealing with XML from friend Jacob Sebastian on Select * FROM XML which he suggests looking at before reading his own advice on the same topic.


We continue to see the traction of MySQL within Oracle communities.  More and more user group and Oracle-focused events are adding MySQL content to their agendas.  At ODTUG/Kaleidoscope, Sheeri Cabral and Ronald Bradford were instrumental in developing the MySQL track of 19 sessions, and are speaking themselves.


Aleksander Kuzminsky announced that Percona Server 5.1.47-rel11 is available for download, noting new features and fixed bugs.


Craig Mullins writes how to access your DB2 Catalog “Poster” online for any of you constantly looking for DB2 Catalog table and column names.


Dave Page does a bit of comparison and gives his opinion on VoltDB, the new database server architected by the ‘father’ of Postgres, Dr Michael Stonebraker.


Until next week.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:17:"The Pythian Group";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:34;a:6:{s:4:"data";s:53:"
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:2:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:53:"Apache And MySQL Monitoring With Bijk On Debian Lenny";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:79:"http://www.howtoforge.com/apache-and-mysql-monitoring-with-bijk-on-debian-lenny";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:79:"http://www.howtoforge.com/apache-and-mysql-monitoring-with-bijk-on-debian-lenny";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:377:"



Apache And MySQL Monitoring With Bijk On Debian Lenny

This tutorial describes how you can monitor your server with the tool
 Bijk. Bijk creates 
online 30 graphs about load, CPU, memory, traffic, Apache, NginX, 
PostreSQL and others with alerts. Bijk can be used on Debian, Ubuntu, 
CentOS, RedHat and Gentoo. In this article I will explain how to install
 Bijk on Debian.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 14:24:22 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:6:"Debian";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:6:"Apache";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:10:"Monitoring";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:898:"





Apache And MySQL Monitoring With Bijk On Debian Lenny



This tutorial describes how you can monitor your server with the tool
 Bijk. Bijk creates 
online 30 graphs about load, CPU, memory, traffic, Apache, NginX, 
PostreSQL and others with alerts. Bijk can be used on Debian, Ubuntu, 
CentOS, RedHat and Gentoo. In this article I will explain how to install
 Bijk on Debian.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:35;a:6:{s:4:"data";s:73:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:51:"ODTUG Kaleidoscope 2010 — Ready… Set… Go-o-o!";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:36:"http://www.pythian.com/news/?p=13587";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:75:"http://www.pythian.com/news/13587/odtug-kaleidoscope-2010-ready-set-go-o-o/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:2922:"It is time… Time for one more very special conference for me. Why special?

I have never been to ODTUG Kaleidoscope before. I always like new conferences — new experience.
Unlike Sheeri, I do not speak! This is one of those rare conferences where I come to slack off, meet old friends and make new ones, go to lots of sessions and actually learn stuff.
Coming from the DBA background, it’s not often that I come to development oriented conferences and I think I should do more of that.
It’s in Washington, DC. I lived there for some time and have number of good friends there. I’m really excited to see them again!

So what am I going to do there? I just arrived and right in time for the Oracle ACE Directors’ briefing that will run for the whole days of Friday. This is a super secret meeting where Oracle’s super secret plans are shared. Nobody can talk about that after this meeting or their tongues are cut off on the spot. For those of you who didn’t realize I’m joking, the ACE Director’s briefing is where Oracle shares the roadmap of its products — some of it is long term strategy and some is about the upcoming releases. There are few things that we are asked not to share in public but, frankly, there is nothing really sensitive. One of the most interesting parts of the briefing are the Q&A moments when all kind of questions get asked (sometimes tough ones) and, to the most parts, gets answered.

It took me the same time to get to the hotel from Dulles International Airport as flying from Ottawa. Oh well, the beauty of DC traffic. Now I’m sitting at the back of the room enjoying the demo of the new APEX 4.0 and quietly having a bite. I see quite a few familiar faces (or rather backs) that I’m about to say hello to (pending the next break).
So what’s ahead? Yesterday, we all received a surprise invitation (well, surprise to me at least) to ACE Directors welcome reception organized by the ODTUG board members. Looking forward to that.
Saturday is the ODTUG Community Service Day where the volunteers help to beautify Ronald H. Brown Middle School in Washington, D.C.
Sunday is ODTUG Kaleidoscope 2010 Symposia and I’m still not sure whether I should select APEX or Performance, Scalability, and Security track. Sunday night is the night of Oracle ACE dinner and I think I managed to miss few ACE program dinners at the past few conferences so I’m really looking forward to this Sunday.
Monday evening, come for the Sundown Sessions which are open discussions moderated by Oracle ACE Directors. I’ll be hanging either in MySQL or Database Development halls. Immediately following that, on my agenda is Oracle ACE Directors Reception — this is your chance to mingle with the bunch of Oracle ACE Directors and other cool folks and abuse some complimentary food and drinks (thanks to OTN).";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 14:01:24 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:7:{i:0;a:5:{s:4:"data";s:17:"Non-Tech Articles";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:6:"Oracle";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:14:"Technical Blog";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:11:"conferences";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:12:"kaleidoscope";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:5:"odtug";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:13:"Washington DC";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:3604:"
It is time… Time for one more very special conference for me. Why special?


    

  1. I have never been to ODTUG Kaleidoscope before. I always like new conferences — new experience.
    2. 

  2. Unlike Sheeri, I do not speak! This is one of those rare conferences where I come to slack off, meet old friends and make new ones, go to lots of sessions and actually learn stuff.
    3. 

  3. Coming from the DBA background, it’s not often that I come to development oriented conferences and I think I should do more of that.
    4. 

  4. It’s in Washington, DC. I lived there for some time and have number of good friends there. I’m really excited to see them again!
    5. 



So what am I going to do there? I just arrived and right in time for the Oracle ACE Directors’ briefing that will run for the whole days of Friday. This is a super secret meeting where Oracle’s super secret plans are shared. Nobody can talk about that after this meeting or their tongues are cut off on the spot. For those of you who didn’t realize I’m joking, the ACE Director’s briefing is where Oracle shares the roadmap of its products — some of it is long term strategy and some is about the upcoming releases. There are few things that we are asked not to share in public but, frankly, there is nothing really sensitive. One of the most interesting parts of the briefing are the Q&A moments when all kind of questions get asked (sometimes tough ones) and, to the most parts, gets answered.



It took me the same time to get to the hotel from Dulles International Airport as flying from Ottawa. Oh well, the beauty of DC traffic. Now I’m sitting at the back of the room enjoying the demo of the new APEX 4.0 and quietly having a bite. I see quite a few familiar faces (or rather backs) that I’m about to say hello to (pending the next break).


So what’s ahead? Yesterday, we all received a surprise invitation (well, surprise to me at least) to ACE Directors welcome reception organized by the ODTUG board members. Looking forward to that.


Saturday is the ODTUG Community Service Day where the volunteers help to beautify Ronald H. Brown Middle School in Washington, D.C.


Sunday is ODTUG Kaleidoscope 2010 Symposia and I’m still not sure whether I should select APEX or Performance, Scalability, and Security track. Sunday night is the night of Oracle ACE dinner and I think I managed to miss few ACE program dinners at the past few conferences so I’m really looking forward to this Sunday.


Monday evening, come for the Sundown Sessions which are open discussions moderated by Oracle ACE Directors. I’ll be hanging either in MySQL or Database Development halls. Immediately following that, on my agenda is Oracle ACE Directors Reception — this is your chance to mingle with the bunch of Oracle ACE Directors and other cool folks and abuse some complimentary food and drinks (thanks to OTN).

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:14:"Alex Gorbachev";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:36;a:6:{s:4:"data";s:48:"
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:53:"About likelihoods (Om sannolikheter, Tage Danielsson)";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:25:"287 at http://openlife.cc";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:85:"http://openlife.cc/blogs/2010/june/about-likelihoods-om-sannolikheter-tage-danielsson";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:1101:"In celebration of Midsummer today, I wanted to post the below monologue on likelihoods (of nuclear powerplant catastrophies), which is a translation of a famous Swedish monologue by Tage Danielsson.

When we at MySQL had joined Sun, one task for me and my Sales Engineer collagues was to travel to Sun offices and educated the huge Sun sales force about MySQL, so they could sell it too. (Basically to tell them about open source, scale-out, reference customers, and most importantly: Don't sell Cluster on your own, call me first.) Being a Telecom Sales Engineer, I was sent to tour the Ericsson account team meeting, the Nokia account team meeting, and for logistical reasons even the Siemens account team meeting that was at the same location as the Nokia team. 

Each meeting had nicer and nicer dinners, but the Ericsson account team meeting in Stockholm was clearly the winner. The dinner was set in the City Hall restaurant (Stadshuskällaren), which is also were they serve the Nobel gala dinners. Our menu was a copy of the 1981 Nobel menu, served on the authentic Nobel porcelain.

read more";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 12:04:34 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:2:{i:0;a:5:{s:4:"data";s:5:"Funny";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1458:"
In celebration of Midsummer today, I wanted to post the below monologue on likelihoods (of nuclear powerplant catastrophies), which is a translation of a famous Swedish monologue by Tage Danielsson.


When we at MySQL had joined Sun, one task for me and my Sales Engineer collagues was to travel to Sun offices and educated the huge Sun sales force about MySQL, so they could sell it too. (Basically to tell them about open source, scale-out, reference customers, and most importantly: Don't sell Cluster on your own, call me first.) Being a Telecom Sales Engineer, I was sent to tour the Ericsson account team meeting, the Nokia account team meeting, and for logistical reasons even the Siemens account team meeting that was at the same location as the Nokia team. 


Each meeting had nicer and nicer dinners, but the Ericsson account team meeting in Stockholm was clearly the winner. The dinner was set in the City Hall restaurant (Stadshuskällaren), which is also were they serve the Nobel gala dinners. Our menu was a copy of the 1981 Nobel menu, served on the authentic Nobel porcelain.



read more

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:11:"Henrik Ingo";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:37;a:6:{s:4:"data";s:78:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:30:"Four short links: 25 June 2010";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:37:"tag:radar.oreilly.com,2010://57.40145";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:99:"http://feedproxy.google.com/~r/oreilly/radar/atom/~3/ykRJ4TMb8-U/four-short-links-25-june-2010.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:2385:"
Membase -- an open-source (Apache 2.0 license) distributed, key-value database management system optimized for storing data behind interactive web applications. These applications must service many concurrent users; creating, storing, retrieving, aggregating, manipulating and presenting data in real-time. Supporting these requirements, membase processes data operations with quasi-deterministic low latency and high sustained throughput. (via Hacker News)
Sergey's Search (Wired) -- Sergey Brin, one of the Google founders, learned he had a gene allele that gave him much higher odds of getting Parkinson's.  His response has been to help medical research, both with money and through 23andme.  Langston decided to see whether the 23andMe Research Initiative might be able to shed some insight on the correlation, so he rang up 23andMe’s Eriksson, and asked him to run a search. In a few minutes, Eriksson was able to identify 350 people who had the mutation responsible for Gaucher’s. A few clicks more and he was able to calculate that they were five times more likely to have Parkinson’s disease, a result practically identical to the NEJM study. All told, it took about 20 minutes. “It would’ve taken years to learn that in traditional epidemiology,” Langston says. “Even though we’re in the Wright brothers early days with this stuff, to get a result so strongly and so quickly is remarkable.”
Startup.gov (YouTube) -- Anil Dash talk at Personal Democracy Forum on applying insights from startups to government.  I hope the more people say this, the greater the odds it'll be acted on.
Open Core Software -- Marten Mickos (ex-MySQL) talks up "open core" (open source base, proprietary extensions) as a way to resolve the conflict of "change the world with open source" and "make money".  Brian Aker disagrees: There has been no successful launch of an open core company that has reached any significant size, especially of the size that Marten hints at in the article.  My take: there are three reasons for open source (freedoms, price, and development scale) and if you close the source to part of your product then the whole product loses those benefits.  If you open source enough that the open source bit has massive momentum, then you probably don't have enough left proprietary to gain huge financial benefit.



   
";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 10:00:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:8:{i:0;a:5:{s:4:"data";s:8:"big data";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:8:"business";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:8:"genomics";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:6:"gov2.0";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:8:"medicine";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:5:"nosql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:11:"open source";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:8:"startups";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:3974:"
    

  1. Membase -- an open-source (Apache 2.0 license) distributed, key-value database management system optimized for storing data behind interactive web applications. These applications must service many concurrent users; creating, storing, retrieving, aggregating, manipulating and presenting data in real-time. Supporting these requirements, membase processes data operations with quasi-deterministic low latency and high sustained throughput. (via Hacker News)
    2. 

  2. Sergey's Search (Wired) -- Sergey Brin, one of the Google founders, learned he had a gene allele that gave him much higher odds of getting Parkinson's.  His response has been to help medical research, both with money and through 23andme.  Langston decided to see whether the 23andMe Research Initiative might be able to shed some insight on the correlation, so he rang up 23andMe’s Eriksson, and asked him to run a search. In a few minutes, Eriksson was able to identify 350 people who had the mutation responsible for Gaucher’s. A few clicks more and he was able to calculate that they were five times more likely to have Parkinson’s disease, a result practically identical to the NEJM study. All told, it took about 20 minutes. “It would’ve taken years to learn that in traditional epidemiology,” Langston says. “Even though we’re in the Wright brothers early days with this stuff, to get a result so strongly and so quickly is remarkable.”
    3. 

  3. Startup.gov (YouTube) -- Anil Dash talk at Personal Democracy Forum on applying insights from startups to government.  I hope the more people say this, the greater the odds it'll be acted on.
    4. 

  4. Open Core Software -- Marten Mickos (ex-MySQL) talks up "open core" (open source base, proprietary extensions) as a way to resolve the conflict of "change the world with open source" and "make money".  Brian Aker disagrees: There has been no successful launch of an open core company that has reached any significant size, especially of the size that Marten hints at in the article.  My take: there are three reasons for open source (freedoms, price, and development scale) and if you close the source to part of your product then the whole product loses those benefits.  If you open source enough that the open source bit has massive momentum, then you probably don't have enough left proprietary to gain huge financial benefit.
    5. 





   


PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:14:"Nat Torkington";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:38;a:6:{s:4:"data";s:58:"
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:47:"Quick start guides: MySQL cluster in 10 minutes";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:59:"tag:blogger.com,1999:blog-16959946.post-4343278648414547351";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:83:"http://datacharmer.blogspot.com/2010/06/quick-start-guides-mysql-cluster-in-10.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:805:" Scared of MySQL Cluster? Don't be. You may want to try the quick start guides that are available in the Cluster downloads page.These guides are a step-by-step instructions set to create a simple cluster in one host.Following the instructions, you will be able to shape up all the cluster components in 10 minutes or less. This will be far from production ready, of course, but it shows that cluster is not rocket science, and anyone can get started with it with a minimal time investment.I tried the Linux instructions on my Mac, and it worked without need for any changes. Things may be different when you deploy a real set of servers on separate hosts, but it's a good start.If I compare this guide with my first experience with MySQL Cluster in 2004, usability and stability have increased enormously.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 09:16:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:5:"mysql";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:7:"cluster";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:13:"documentation";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:8:"tutorial";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1563:"
MySQL Cluster quick start guide Scared of MySQL Cluster? 
Don't be. You may want to try the quick start guides that are available in the Cluster downloads page.
These guides are a step-by-step instructions set to create a simple cluster in one host.
Following the instructions, you will be able to shape up all the cluster components in 10 minutes or less. This will be far from production ready, of course, but it shows that cluster is not rocket science, and anyone can get started with it with a minimal time investment.
I tried the Linux instructions on my Mac, and it worked without need for any changes. Things may be different when you deploy a real set of servers on separate hosts, but it's a good start.
If I compare this guide with my first experience with MySQL Cluster in 2004, usability and stability have increased enormously.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:14:"Giuseppe Maxia";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:39;a:6:{s:4:"data";s:58:"
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:76:"Download, install, configure, run and test MySQL Cluster in under 15 minutes";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:32:"http://www.clusterdb.com/?p=1125";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:105:"http://www.clusterdb.com/mysql/download-install-configure-run-and-test-mysql-cluster-in-under-15-minutes/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:869:"Single host Cluster
A series of quick-start guides are now available to get you up and running with MySQL Cluster in as little time as possible; they are available for LINUX/Mac OS X, Windows and Solaris. The configuration is intentionally a simple one – 2 data nodes, 1 management node and 1 MySQL Server. Once you have this up and running, your next experiment may be to extend this over multiple hosts.
Download the Quick Start Guide for your platform below:

MySQL Cluster Quick Start for Linux – also works for Mac OS X
MySQL Cluster Quick Start for Windows
MySQL Cluster Quick Start for Solaris

These links are also available from the MySQL Cluster download page.
The intent is that these guides should be simple to follow even if you have never tried MySQL Cluster (or even MySQL) before. Any comments or suggested improvements would be appreciated.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 08:18:07 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:5:"MySQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:6:"Instal";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:9:"MySQL CGE";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:13:"MySQL Cluster";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1884:"
Single host Cluster


A series of quick-start guides are now available to get you up and running with MySQL Cluster in as little time as possible; they are available for LINUX/Mac OS X, Windows and Solaris. The configuration is intentionally a simple one – 2 data nodes, 1 management node and 1 MySQL Server. Once you have this up and running, your next experiment may be to extend this over multiple hosts.


Download the Quick Start Guide for your platform below:



These links are also available from the MySQL Cluster download page.


The intent is that these guides should be simple to follow even if you have never tried MySQL Cluster (or even MySQL) before. Any comments or suggested improvements would be appreciated.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:13:"Andrew Morgan";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:40;a:6:{s:4:"data";s:38:"
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:22:"dsh and TABLE CHECKSUM";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:59:"tag:blogger.com,1999:blog-31421954.post-1663233179474927611";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:64:"http://mysqldba.blogspot.com/2010/06/dsh-and-table-checksum.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:2801:"So running through some various tasks, I'm finally on the section of work where I can resurrect a script that finds inconsistent data between master-master pairs. Let's get a quick summary to find our problems to target the script at.dsh -ef 20 -w dbfacebook9b,dbfacebook9a -s ~/bin/execute_query.sh "FacebookShard \"CHECKSUM TABLE RollingUserLog\""dbfacebook9b: *************************** 1. row ***************************dbfacebook9b:    Table: FacebookShard.RollingUserLogdbfacebook9b: Checksum: 538386033dbfacebook9a: *************************** 1. row ***************************dbfacebook9a:    Table: FacebookShard.RollingUserLogdbfacebook9a: Checksum: 538386033not bad.CHECKSUM TABLE uses a ACCUM algorithm to determine the table checksum. This gives a good fast snapshot in a shared lock mode so writes do not stop while doing a checksum (this is in the context of INNODB). RollingUserLog gets nearly 700 writes per sec per box (No alerts).But what if there is a problem# dsh -ef 20 -w dbfacebook38b,dbfacebook38a -s ~/bin/execute_query.sh "FacebookShard \"CHECKSUM TABLE SimulatedProblem\""dbfacebook38b: *************************** 1. row ***************************dbfacebook38b:    Table: FacebookShard.SimulatedProblemdbfacebook38b: Checksum: 660032421dbfacebook38a: *************************** 1. row ***************************dbfacebook38a:    Table: FacebookShard.SimulatedProblemdbfacebook38a: Checksum: 2533654621Well let's see if there is a row count mismatch# dsh -ef 20 -w dbfacebook38b,dbfacebook38a -s ~/bin/execute_query.sh "FacebookShard \"SELECT COUNT(*) FROM SimulatedProblem\""dbfacebook38b: *************************** 1. row ***************************dbfacebook38b: COUNT(*): 64358dbfacebook38a: *************************** 1. row ***************************dbfacebook38a: COUNT(*): 64358Ah so the data is inconsistent, we are not missing rows but 1 or more rows have different values. Now time to find it.To resurrect my script now that I know what table is messed up:Algorithm is this:open connection to both servers - forkcompare the data by scanning the table via a index walkcrc32/md5/hash the rowscompare columns if (dbfacebook38b.hash != dbfacebook38a.hash)mark row and record primary key to track positionprint reportbut wait why polish up my script when I can use a formal one that does the job great!@seemk-table-checksummk-table-sync# fixes the issuemk-table-sync h=dbfacebook38a,u=$USER,p=$PASS,D=FacebookShard,t=SimulatedProblem h=dbfacebook38b --execute --no-check-slavemaatkit is awesome. Nearly every problem that I run into and built a script for has been formalized. I'm going to muster up some time and contribute to this toolkit like add my binary log rotate which makes sure that the slaves are caught up to the binary file that is being purged.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 08:02:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:3611:"So running through some various tasks, I'm finally on the section of work where I can resurrect a script that finds inconsistent data between master-master pairs. 

Let's get a quick summary to find our problems to target the script at.


dsh -ef 20 -w dbfacebook9b,dbfacebook9a -s ~/bin/execute_query.sh "FacebookShard \"CHECKSUM TABLE RollingUserLog\""
dbfacebook9b: *************************** 1. row ***************************
dbfacebook9b:    Table: FacebookShard.RollingUserLog
dbfacebook9b: Checksum: 538386033
dbfacebook9a: *************************** 1. row ***************************
dbfacebook9a:    Table: FacebookShard.RollingUserLog
dbfacebook9a: Checksum: 538386033


not bad.

CHECKSUM TABLE uses a ACCUM algorithm to determine the table checksum. This gives a good fast snapshot in a shared lock mode so writes do not stop while doing a checksum (this is in the context of INNODB). RollingUserLog gets nearly 700 writes per sec per box (No alerts).

But what if there is a problem

# dsh -ef 20 -w dbfacebook38b,dbfacebook38a -s ~/bin/execute_query.sh "FacebookShard \"CHECKSUM TABLE SimulatedProblem\""
dbfacebook38b: *************************** 1. row ***************************
dbfacebook38b:    Table: FacebookShard.SimulatedProblem
dbfacebook38b: Checksum: 660032421
dbfacebook38a: *************************** 1. row ***************************
dbfacebook38a:    Table: FacebookShard.SimulatedProblem
dbfacebook38a: Checksum: 2533654621


Well let's see if there is a row count mismatch


# dsh -ef 20 -w dbfacebook38b,dbfacebook38a -s ~/bin/execute_query.sh "FacebookShard \"SELECT COUNT(*) FROM SimulatedProblem\""
dbfacebook38b: *************************** 1. row ***************************
dbfacebook38b: COUNT(*): 64358
dbfacebook38a: *************************** 1. row ***************************
dbfacebook38a: COUNT(*): 64358



Ah so the data is inconsistent, we are not missing rows but 1 or more rows have different values. Now time to find it.

To resurrect my script now that I know what table is messed up:

Algorithm is this:

open connection to both servers - fork
compare the data by scanning the table via a index walk
crc32/md5/hash the rows
compare columns 
if (dbfacebook38b.hash != dbfacebook38a.hash)
mark row and record primary key to track position

print report

but wait why polish up my script when I can use a formal one that does the job great!

@see

mk-table-checksum
mk-table-sync

# fixes the issue

mk-table-sync h=dbfacebook38a,u=$USER,p=$PASS,D=FacebookShard,t=SimulatedProblem h=dbfacebook38b --execute --no-check-slave


maatkit is awesome. Nearly every problem that I run into and built a script for has been formalized. I'm going to muster up some time and contribute to this toolkit like add my binary log rotate which makes sure that the slaves are caught up to the binary file that is being purged.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:17:"Dathan Pattishall";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:41;a:6:{s:4:"data";s:58:"
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:24:"OpenSQL Camp Boston 2010";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:33:"http://www.xaprb.com/blog/?p=1900";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:62:"http://www.xaprb.com/blog/2010/06/25/opensql-camp-boston-2010/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:407:"Sheeri and others are organizing another incarnation of OpenSQL Camp in October in Boston.  You ought to go!  It’s relevant to MySQL, PostgreSQL, SQLite, and lots of the newer generation of databases — MongoDB, Cassandra, and so on.

Related posts:Going to OpenSQL Camp US 2009OpenSQL Camp events in 2009The history of OpenSQL CampRecap of Portland OpenSQL Camp 2009OpenSQL Camp badges are ready";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 06:23:01 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:11:"Conferences";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:12:"OpenSQL Camp";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:3:"SQL";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:21:"Sheeri Kritzer Cabral";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:1503:"
Sheeri and others are organizing another incarnation of OpenSQL Camp in October in Boston.  You ought to go!  It’s relevant to MySQL, PostgreSQL, SQLite, and lots of the newer generation of databases — MongoDB, Cassandra, and so on.



Related posts:
  1. Going to OpenSQL Camp US 2009
  2. OpenSQL Camp events in 2009
  3. The history of OpenSQL Camp
  4. Recap of Portland OpenSQL Camp 2009
  5. OpenSQL Camp badges are ready

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:22:"Baron Schwartz (xaprb)";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:42;a:6:{s:4:"data";s:33:"
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:2:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:44:"Using PMP to double MySQL throughput, part 1";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:53:"http://www.facebook.com/note.php?note_id=404965725932";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:53:"http://www.facebook.com/note.php?note_id=404965725932";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:5064:"I used PMP this week to find bottlenecks in MySQL while running two benchmarks and describe the first bottleneck in this note. Changing the code as described here doubled the peak QPS. Pardon the hype in the title. This doubles peak QPS at 64 and 128 concurrent threads. To double throughput in all cases you might need faster CPU cores and a faster network. I created feature request 54790 for this.

I ran sysbench with concurrent threads that fetch one row by primary key per statement. HANDLER was used instead of SELECT. This is a workload for which some NoSQL servers are very fast. The sysbench client ran on one host and the mysqld server on another. Throughput saturated at less than 80,000 QPS for more than 32 concurrent threads. On previous tests with the sysbench client and mysqld server running on the same host I have been able to get more than 150,000 QPS.

PMP was used on the mysqld host to determine whether there was an obvious bottleneck and by far the most common thread stack was:

fcntl,vio_blocking,my_real_read,my_net_read,do_command,handle_one_connection,start_thread,clone

The possible source for fcntl has a call to setfl from the code that handles fcntl and setfl calls lock_kernel. That might explain the bottleneck.

It is possible to call fcntl less frequently. The calls come from vio_blocking which is called by my_real_read in net_serv.cc. The sockets used by mysqld are put in non-blocking mode (O_NONBLOCK). When reading data mysqld first attempts a read in non-blocking mode and if the read gets a timeout then it puts the socket in blocking mode via a call to fcntl and retries the read. At the end of my_real_read an additional call to vio_blocking (and fcntl) can be done to return the socket to non-blocking mode.

The alternative is to keep the socket in blocking mode, avoid the frequent calls to fcntl and rely on SO_SNDTIMEO and SO_RCVTIMEO. To test this, I modified mysqld to keep sockets in blocking mode. The diff is simple and sufficient for performance testing, but I suspect more code needs to be changed:

--- original/sql/net_serv.cc     2010-06-16 11:44:33.000000000 -0700
+++ new/sql/net_serv.cc      2010-06-24 12:58:07.000000000 -0700
@@ -153,7 +153,7 @@
     if (!(test_flags & TEST_BLOCKING))
     {
       my_bool old_mode;
-      vio_blocking(vio, FALSE, &old_mode);
+      vio_blocking(vio, TRUE, &old_mode);
     }
 #endif
     vio_fastsend(vio);

I ran sysbench on a server with 16 cores and Linux 2.6 using 3 binaries: 5084forever is 5.0.84 with the Facebook patch, 5147orig is unmodified 5.1.47 and 5147fcntl is 5.1.47 with the Facebook patch, the diff above and a few other changes that will be described in the next note. The significant change is the diff above. The results below list the QPS where each query is a HANDLER statement that fetches one row by primary key. The results are listed for 1, 2, 4, 8, 16, 32, 64, 128 and 256 concurrent threads.With the diff above the peak QPS is more than doubled. This is a graph of the results.

  1042   2053   4377   9784  24994  81115 174279 183676 108874      5147fcntl
  1093   2169   4343   9419  26350  80732  85759  82283  76116      5084forever
   952   2010   4281   9954  25239  77551  82038  80430  79439      5147orig

This lists 95th percentile response time in microseconds as reported by sysbench:

  1990   1880   1250   1160   1090    730    550   1950   7430      5147fcntl
  1130   1190   1250   1140   1070    750   1400   3130   6850      5084forever
  2010   1940   1430   1130   1080    700   1430   3210   6790      5147orig


sysbench was run using this command line. The sbtest table has 8m rows. This uses HANDLER to fetch one row by primary key. The innodb buffer pool was large enough to cache all data and the adaptive hash index was primed before the test was run.

# run for $nt in 1, 2, 4, ... 256
sysbench --test=oltp --oltp-table-size=8000000 --max-time=180 --max-requests=0 \
  --mysql-table-engine=innodb --db-ps-mode=disable --mysql-engine-trx=yes \
  --oltp-read-only --oltp-skip-trx --oltp-test-mode=simple \
  --oltp-point-select-mysql-handler --oltp-dist-type=uniform \
  --oltp-range-size=1 --num-threads=$nt --seed-rng=$nt run

These values were used for my.cnf:

innodb_buffer_pool_size=4000M
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
max_connections=2000
table_cache=2000

The configure command lines used to build the 5.1 binaries.

# 5147orig
configure --enable-thread-safe-client \
  --with-plugins=partition,csv,blackhole,myisam,heap,innodb_plugin \
  --without-plugin-innobase --with-fast-mutexes --with-extra-charsets=all \
  --without-debug \
  C_EXTRA_FLAGS="-fno-omit-frame-pointer -fno-strict-aliasing -DNO_ALARM -DSIGNAL_WITH_VIO_CLOSE -Wall"

# 5147fcntl
configure --without-debug --enable-thread-safe-client \
  --with-plugins=partition,csv,blackhole,myisam,heap,innodb_plugin,pbxt \
  --without-plugin-innobase --with-fast-mutexes --with-extra-charsets=all\
   C_EXTRA_FLAGS="-g -O2 -fno-omit-frame-pointer -fno-strict-aliasing -DNO_ALARM -DSIGNAL_WITH_VIO_CLOSE -Wall"
";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 03:54:13 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:6445:"I used PMP this week to find bottlenecks in MySQL while running two benchmarks and describe the first bottleneck in this note. Changing the code as described here doubled the peak QPS. Pardon the hype in the title. This doubles peak QPS at 64 and 128 concurrent threads. To double throughput in all cases you might need faster CPU cores and a faster network. I created feature request 54790 for this.

I ran sysbench with concurrent threads that fetch one row by primary key per statement. HANDLER was used instead of SELECT. This is a workload for which some NoSQL servers are very fast. The sysbench client ran on one host and the mysqld server on another. Throughput saturated at less than 80,000 QPS for more than 32 concurrent threads. On previous tests with the sysbench client and mysqld server running on the same host I have been able to get more than 150,000 QPS.

PMP was used on the mysqld host to determine whether there was an obvious bottleneck and by far the most common thread stack was:

fcntl,vio_blocking,my_real_read,my_net_read,do_command,handle_one_connection,start_thread,clone


The possible source for fcntl has a call to setfl from the code that handles fcntl and setfl calls lock_kernel. That might explain the bottleneck.

It is possible to call fcntl less frequently. The calls come from vio_blocking which is called by my_real_read in net_serv.cc. The sockets used by mysqld are put in non-blocking mode (O_NONBLOCK). When reading data mysqld first attempts a read in non-blocking mode and if the read gets a timeout then it puts the socket in blocking mode via a call to fcntl and retries the read. At the end of my_real_read an additional call to vio_blocking (and fcntl) can be done to return the socket to non-blocking mode.

The alternative is to keep the socket in blocking mode, avoid the frequent calls to fcntl and rely on SO_SNDTIMEO and SO_RCVTIMEO. To test this, I modified mysqld to keep sockets in blocking mode. The diff is simple and sufficient for performance testing, but I suspect more code needs to be changed:

--- original/sql/net_serv.cc     2010-06-16 11:44:33.000000000 -0700
+++ new/sql/net_serv.cc      2010-06-24 12:58:07.000000000 -0700
@@ -153,7 +153,7 @@
     if (!(test_flags & TEST_BLOCKING))
     {
       my_bool old_mode;
-      vio_blocking(vio, FALSE, &old_mode);
+      vio_blocking(vio, TRUE, &old_mode);
     }
 #endif
     vio_fastsend(vio);


I ran sysbench on a server with 16 cores and Linux 2.6 using 3 binaries: 5084forever is 5.0.84 with the Facebook patch, 5147orig is unmodified 5.1.47 and 5147fcntl is 5.1.47 with the Facebook patch, the diff above and a few other changes that will be described in the next note. The significant change is the diff above. The results below list the QPS where each query is a HANDLER statement that fetches one row by primary key. The results are listed for 1, 2, 4, 8, 16, 32, 64, 128 and 256 concurrent threads.With the diff above the peak QPS is more than doubled. This is a graph of the results.

  1042   2053   4377   9784  24994  81115 174279 183676 108874      5147fcntl
  1093   2169   4343   9419  26350  80732  85759  82283  76116      5084forever
   952   2010   4281   9954  25239  77551  82038  80430  79439      5147orig


This lists 95th percentile response time in microseconds as reported by sysbench:

  1990   1880   1250   1160   1090    730    550   1950   7430      5147fcntl
  1130   1190   1250   1140   1070    750   1400   3130   6850      5084forever
  2010   1940   1430   1130   1080    700   1430   3210   6790      5147orig



sysbench was run using this command line. The sbtest table has 8m rows. This uses HANDLER to fetch one row by primary key. The innodb buffer pool was large enough to cache all data and the adaptive hash index was primed before the test was run.

# run for $nt in 1, 2, 4, ... 256
sysbench --test=oltp --oltp-table-size=8000000 --max-time=180 --max-requests=0 \
  --mysql-table-engine=innodb --db-ps-mode=disable --mysql-engine-trx=yes \
  --oltp-read-only --oltp-skip-trx --oltp-test-mode=simple \
  --oltp-point-select-mysql-handler --oltp-dist-type=uniform \
  --oltp-range-size=1 --num-threads=$nt --seed-rng=$nt run


These values were used for my.cnf:

innodb_buffer_pool_size=4000M
innodb_flush_method=O_DIRECT
innodb_thread_concurrency=0
max_connections=2000
table_cache=2000


The configure command lines used to build the 5.1 binaries.

# 5147orig
configure --enable-thread-safe-client \
  --with-plugins=partition,csv,blackhole,myisam,heap,innodb_plugin \
  --without-plugin-innobase --with-fast-mutexes --with-extra-charsets=all \
  --without-debug \
  C_EXTRA_FLAGS="-fno-omit-frame-pointer -fno-strict-aliasing -DNO_ALARM -DSIGNAL_WITH_VIO_CLOSE -Wall"

# 5147fcntl
configure --without-debug --enable-thread-safe-client \
  --with-plugins=partition,csv,blackhole,myisam,heap,innodb_plugin,pbxt \
  --without-plugin-innobase --with-fast-mutexes --with-extra-charsets=all\
   C_EXTRA_FLAGS="-g -O2 -fno-omit-frame-pointer -fno-strict-aliasing -DNO_ALARM -DSIGNAL_WITH_VIO_CLOSE -Wall"


PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:43;a:6:{s:4:"data";s:33:"
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:2:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:49:"More updates for the Facebook patch for MySQL 5.1";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:53:"http://www.facebook.com/note.php?note_id=404953860932";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:53:"http://www.facebook.com/note.php?note_id=404953860932";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:1355:"I pushed more patches to the Facebook patch for MySQL 5.1 at Launchpad. Alas, I have yet to publish release notes. The patches include:

Merged to MySQL 5.1.47
Backports from future 5.1 releases of bug fixes
More background disk IO counted in information_schema.table_statistics
Add the innodb_fast_checksums option that uses crc32 and is 10X faster
Provide an option to do fuzzy checkpoint IO in the main background thread. On some servers we get many stalls for fuzzy checkpoint IO in user threads. This can be a big source of replication latency when the stalled thread is the slave SQL thread. It is easy to overlook this problem if your MySQL binary doesn't report time in the function log_checkpoint_margin. The Facebook patch reports that in SHOW INNODB STATUS and SHOW GLOBAL STATUS.
Suppress or fix bogus valgrind warnings generated by mysql-test-run. Most of these are fixed in future releases. But I want to run 5.1.47 and be confident there are no real valgrind warnings.
Add innodb_max_pct_of_buffer_pool to set a soft limit on the max space in the buffer pool to be used for the insert buffer. The hard max and without this change the only limit is 50% and I think that is too much. Also add innodb_max_pct_of_io_capacity to set the percentage of innodb_io_capacity to be used for insert buffer merges when the size limit has been exceeded.
";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Fri, 25 Jun 2010 01:41:43 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:2100:"I pushed more patches to the Facebook patch for MySQL 5.1 at Launchpad. Alas, I have yet to publish release notes. The patches include:

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:44;a:6:{s:4:"data";s:58:"
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:34:"Sure-fire MySQL Install on Windows";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:36:"http://lstigile.wordpress.com/?p=123";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:76:"http://lstigile.wordpress.com/2010/06/24/sure-fire-mysql-install-on-windows/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:3898:"Due to firewalls, virus scans, corporate security restrictions, or just plain bad luck; there are times when MySQL just won’t install on a specific Windows server.  Here’s a sure-fire install method.   Relax, I won’t have you run the msi installer again.
The first step is to remove directories from your previous install attempts.  Uninstall from the control panel.  Manually, rename or delete “C:\Program Files\MySQL\MySQL Server 5.1″.  Next, delete (or rename) the data directory.  Warning! If you have previously entered data into MySQL, deleting the data directory will delete data.  The directory is located at “C:\Documents and Settings\All Users\Application Data\MySQL\MySQL Server 5.1\data”.   You may need to unhide the data directory.
Let’s get started.
#1.  Download the no-install MySQL Version for Windows.  Choose either “Windows (x86, 32-bit) ZIP Archive” or “Windows (x86, 64-bit) ZIP Archive” based on whether your system is 64 or 32-bit .
#2.  Unzip the files to “C:\Program Files\MySQL\MySQL Server 5.1″.
#3.  Use notepad or wordpad to create a my.ini file and save the file in the MySQL directory, “C:\Program Files\MySQL\MySQL Server 5.1″   Be careful that notepad doesn’t add a default “.txt” extension to the my.ini file ( http://support.microsoft.com/kb/253688 )
Add the following following to your my.ini file and save.

[mysqld]
##update basedir if you move the install location
basedir="C:/Program Files/MySQL/MySQL Server 5.1"

##update datadir if you move the data directory
datadir="C:/Program Files/MySQL/MySQL Server 5.1/data"

## increase to 40% of RAM if using MyISAM
key_buffer_size=32MB

## increase to 80% of available RAM for production usage
innodb_buffer_pool_size = 128MB

## may want to increase if write intensive
innodb_log_file_size = 64MB
innodb_log_buffer_size=8MB
table_cache=1024
thread_cache=16
query_cache_size=32M

Note the basedir and the datadir variables.  If you wish to install in a different location or put the data in a different location, update these values accordingly.
#3.  Create and start the service.    Open a command prompt, and enter the following:

"C:\Program Files\MySQL\MySQL Server 5.1\bin\mysqld"  --install "MySQL 5.1" --defaults-file="C:\Program Files\MySQL\MySQL Server 5.1\my.ini"

This installs the server.  To start the service, type

net start "MySQL 5.1"

or start the service from the control panel.
You should be up and running.  If not, check your err log in “C:\Program Files\MySQL\MySQL Server 5.1\Data”
#4.  Connect from your client application and update your user accounts.  By default, there are 3 accounts: root@localhost, root@127.0.0.1 and an anonymous account.
The following commands log in with the mysql command-line client, remove the anonymous account and set the password for the root accounts.
At a command prompt, enter the following:

"c:\program files\mysql\mysql 5.1 server\bin\mysql" -uroot

This will log you into MySQL and you should have a mysql command prompt like mysql>
Here are the commands to delete your anonymous account and set passwords for your two root accounts.  When setting the password, substitute your desired password for “mypassword”.


mysql> DROP USER ""@localhost;
mysql> Set PASSWORD FOR "root"@"localhost" = password("mypassword");
mysql> Set PASSWORD FOR "root"@"127.0.0.1" = password("mypassword");
mysql> flush privileges;
mysql> exit

Feel free to post any questions/comments or issues to the comment section of this blog.  For more comprehensive instructions see http://dev.mysql.com/doc/refman/5.1/en/windows-install-archive.html
       ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Thu, 24 Jun 2010 22:07:27 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:4:{i:0;a:5:{s:4:"data";s:13:"MySQL Install";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:7:"Windows";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:7:"install";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:15:"Windows Install";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:5720:"
Due to firewalls, virus scans, corporate security restrictions, or just plain bad luck; there are times when MySQL just won’t install on a specific Windows server.  Here’s a sure-fire install method.   Relax, I won’t have you run the msi installer again.


The first step is to remove directories from your previous install attempts.  Uninstall from the control panel.  Manually, rename or delete “C:\Program Files\MySQL\MySQL Server 5.1″.  Next, delete (or rename) the data directory.  Warning! If you have previously entered data into MySQL, deleting the data directory will delete data.  The directory is located at “C:\Documents and Settings\All Users\Application Data\MySQL\MySQL Server 5.1\data”.   You may need to unhide the data directory.


Let’s get started.


#1.  Download the no-install MySQL Version for Windows.  Choose either “Windows (x86, 32-bit) ZIP Archive” or “Windows (x86, 64-bit) ZIP Archive” based on whether your system is 64 or 32-bit .


#2.  Unzip the files to “C:\Program Files\MySQL\MySQL Server 5.1″.


#3.  Use notepad or wordpad to create a my.ini file and save the file in the MySQL directory, “C:\Program Files\MySQL\MySQL Server 5.1″   Be careful that notepad doesn’t add a default “.txt” extension to the my.ini file ( http://support.microsoft.com/kb/253688 )


Add the following following to your my.ini file and save.


[mysqld]
##update basedir if you move the install location
basedir="C:/Program Files/MySQL/MySQL Server 5.1"

##update datadir if you move the data directory
datadir="C:/Program Files/MySQL/MySQL Server 5.1/data"

## increase to 40% of RAM if using MyISAM
key_buffer_size=32MB

## increase to 80% of available RAM for production usage
innodb_buffer_pool_size = 128MB

## may want to increase if write intensive
innodb_log_file_size = 64MB
innodb_log_buffer_size=8MB
table_cache=1024
thread_cache=16
query_cache_size=32M



Note the basedir and the datadir variables.  If you wish to install in a different location or put the data in a different location, update these values accordingly.


#3.  Create and start the service.    Open a command prompt, and enter the following:


"C:\Program Files\MySQL\MySQL Server 5.1\bin\mysqld"  --install "MySQL 5.1" --defaults-file="C:\Program Files\MySQL\MySQL Server 5.1\my.ini"



This installs the server.  To start the service, type


net start "MySQL 5.1"



or start the service from the control panel.


You should be up and running.  If not, check your err log in “C:\Program Files\MySQL\MySQL Server 5.1\Data”


#4.  Connect from your client application and update your user accounts.  By default, there are 3 accounts: root@localhost, root@127.0.0.1 and an anonymous account.


The following commands log in with the mysql command-line client, remove the anonymous account and set the password for the root accounts.


At a command prompt, enter the following:


"c:\program files\mysql\mysql 5.1 server\bin\mysql" -uroot



This will log you into MySQL and you should have a mysql command prompt like mysql>


Here are the commands to delete your anonymous account and set passwords for your two root accounts.  When setting the password, substitute your desired password for “mypassword”.



mysql> DROP USER ""@localhost;
mysql> Set PASSWORD FOR "root"@"localhost" = password("mypassword");
mysql> Set PASSWORD FOR "root"@"127.0.0.1" = password("mypassword");
mysql> flush privileges;
mysql> exit



Feel free to post any questions/comments or issues to the comment section of this blog.  For more comprehensive instructions see http://dev.mysql.com/doc/refman/5.1/en/windows-install-archive.html


       
PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:11:"Lee Stigile";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:45;a:6:{s:4:"data";s:38:"
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:47:"Doing your own on-time flight analysis, Part II";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:70:"tag:blogger.com,1999:blog-8575059197193667898.post-1734685454458516409";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:86:"http://dave-stokes.blogspot.com/2010/06/doing-your-own-on-time-flight-analysis_24.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:2796:"The On Time Flight Data (see previous entry) has 93 rows per flight and the first line of the CSV file has the names of the columns.  Use head -1 on that file to obtain that information and to build a schema by piping the output to a file (and do not forget to remove that top line before loading the data).  "Year", "Quarter", "Month", "DayofMonth", ..."Div5LongestGTime", "Div5WheelsOff", "Div5TailNum",Strip out the quotation marks, add CREATE TABLE flight ( at the front and ) ENGINE=InfiniDB; at the end. Year is the title of the first column and it has been renamed to FlightYear to avoid confusion with the function of the same name. The next part is to assign a data type to each column and a quick perusal of the data will show which fields are numeric,  which ones are alphanumeric, and the one column of type DATE.  Personal preference on the designation are left up to the reader as an exercise in data architecture.  CREATE TABLE flight (FlightYear smallint, Quarter tinyint,Month tinyint,...Div5WheelsOff int,Div5TailNum  int) engine=InfiniDB;Create the table by running MySQL and sourcing the file created.  Now the data needs some attention.The InfiniDB bulk loader cpimport can use any character to delimit fields. In CSV files it is painfully obvious that the fields are separated by commas.  However the On Time Flight Data has a column, OriginCityName, that contains a comma.  A extra column can be added after OriginCityName, the comma can be removed from that one column, OR the other commas need to be changed to another character.  The easiest for most will be adding an extra column.Next a XML file needs to be created using colxml, colxml -j 50 -d "," -l On_Time_Performance_2010_1.csv -t flight air. The the data can be loaded with cpimport, cpimport -j 50.So for a quick test!  Can I find the flights from LA back to Texas?mysql> SELECT FlightNum, COUNT(FlightNum) FROM flight WHERE Origin = 'LAX' and Dest = 'DFW' GROUP BY FlightNum ORDER BY FlightNum;+-----------+------------------+| FlightNum | COUNT(FlightNum) |+-----------+------------------+|      2400 |               31 ||      2404 |               30 ||      2410 |               31 ||      2412 |               31 ||      2416 |               31 ||      2422 |               31 ||      2428 |               31 ||      2434 |               31 ||      2436 |               31 ||      2440 |               31 ||      2444 |               21 ||      2446 |               31 ||      2448 |               31 ||      2450 |               26 ||      2464 |               31 ||      2470 |               31 ||      2501 |                1 ||      6219 |               31 ||      6234 |               31 |+-----------+------------------+19 rows in set (0.52 sec)Next time, find the flight least likely to be delayed!";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Thu, 24 Jun 2010 17:29:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:3725:"The On Time Flight Data (see previous entry) has 93 rows per flight and the first line of the CSV file has the names of the columns.  Use head -1 on that file to obtain that information and to build a schema by piping the output to a file (and do not forget to remove that top line before loading the data).  

"Year", "Quarter", "Month", "DayofMonth", 
...
"Div5LongestGTime", "Div5WheelsOff", "Div5TailNum",

Strip out the quotation marks, add CREATE TABLE flight ( at the front and ) ENGINE=InfiniDB; at the end. Year is the title of the first column and it has been renamed to FlightYear to avoid confusion with the function of the same name. The next part is to assign a data type to each column and a quick perusal of the data will show which fields are numeric,  which ones are alphanumeric, and the one column of type DATE.  Personal preference on the designation are left up to the reader as an exercise in data architecture.  

CREATE TABLE flight (FlightYear smallint, 
Quarter tinyint,
Month tinyint,
...
Div5WheelsOff int,
Div5TailNum  int) engine=InfiniDB;

Create the table by running MySQL and sourcing the file created.  Now the data needs some attention.

The InfiniDB bulk loader cpimport can use any character to delimit fields. In CSV files it is painfully obvious that the fields are separated by commas.  However the On Time Flight Data has a column, OriginCityName, that contains a comma.  A extra column can be added after OriginCityName, the comma can be removed from that one column, OR the other commas need to be changed to another character.  The easiest for most will be adding an extra column.

Next a XML file needs to be created using colxml, colxml -j 50 -d "," -l On_Time_Performance_2010_1.csv -t flight air. The the data can be loaded with cpimport, cpimport -j 50.

So for a quick test!  Can I find the flights from LA back to Texas?


mysql> SELECT FlightNum, COUNT(FlightNum) FROM flight WHERE Origin = 'LAX' and Dest = 'DFW' GROUP BY FlightNum ORDER BY FlightNum;
+-----------+------------------+
| FlightNum | COUNT(FlightNum) |
+-----------+------------------+
|      2400 |               31 |
|      2404 |               30 |
|      2410 |               31 |
|      2412 |               31 |
|      2416 |               31 |
|      2422 |               31 |
|      2428 |               31 |
|      2434 |               31 |
|      2436 |               31 |
|      2440 |               31 |
|      2444 |               21 |
|      2446 |               31 |
|      2448 |               31 |
|      2450 |               26 |
|      2464 |               31 |
|      2470 |               31 |
|      2501 |                1 |
|      6219 |               31 |
|      6234 |               31 |
+-----------+------------------+
19 rows in set (0.52 sec)



Next time, find the flight least likely to be delayed!

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:11:"Dave Stokes";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:46;a:6:{s:4:"data";s:43:"
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:32:"New syslog module for Drizzle DB";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:70:"tag:blogger.com,1999:blog-4771631636333583067.post-5116480211914397223";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:101:"http://feedproxy.google.com/~r/MachinesPlusMinds/~3/i8MwkUP7Q6I/new-syslog-module-for-drizzle-db.html";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:1590:"One of the long-time continuing complaints against using MySQL in an enterprise or large scale environment is it's logging system.  The query log, slow query log, and error log all get written to the server's local filesystem, which is difficult and awkward to hook up to enterprise log collection and analysis systems. A recurring feature request is to be able to use the UNIX syslog instead.

One of the first things I did to Drizzle was tear out the existing logging code. This also removed a number of mutex locks, one of which was completely unnecessary, and one that got constantly hit even if logging was turned off.

In it's place I put two of the first Drizzle plugin interfaces, one for the query logging, which combined the features of the old MySQL query log and the slow query log, and one for error message reporting.  I then wrote a plugin that took the query log, and sent it to the syslog.

At the last MySQL Conference, during the Drizzle Developer Day, Monty Taylor pointed out that we were missing something.  There was no "send error messages to the syslog" plugin. Oops.

Now there is.

I've completely replaced the plugin/logging_syslog module with a new plugin/syslog module that contains three plugins. In addition to a rewritten "query log to syslog", there is "error messages to syslog", and also a new SQL function SYSLOG(facility, priority, text).

I've pushed the work up to launchpad, and I'm confident that after review and possibly some tweaks, it will be integrated with Drizzle.

Documentation and examples for using this new module are in upcoming posts.";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Thu, 24 Jun 2010 15:49:00 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:1:{i:0;a:5:{s:4:"data";s:7:"drizzle";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:2183:"One of the long-time continuing complaints against using MySQL in an enterprise or large scale environment is it's logging system.  The query log, slow query log, and error log all get written to the server's local filesystem, which is difficult and awkward to hook up to enterprise log collection and analysis systems. A recurring feature request is to be able to use the UNIX syslog instead.



One of the first things I did to Drizzle was tear out the existing logging code. This also removed a number of mutex locks, one of which was completely unnecessary, and one that got constantly hit even if logging was turned off.



In it's place I put two of the first Drizzle plugin interfaces, one for the query logging, which combined the features of the old MySQL query log and the slow query log, and one for error message reporting.  I then wrote a plugin that took the query log, and sent it to the syslog.



At the last MySQL Conference, during the Drizzle Developer Day, Monty Taylor pointed out that we were missing something.  There was no "send error messages to the syslog" plugin. Oops.



Now there is.



I've completely replaced the plugin/logging_syslog module with a new plugin/syslog module that contains three plugins. In addition to a rewritten "query log to syslog", there is "error messages to syslog", and also a new SQL function SYSLOG(facility, priority, text).



I've pushed the work up to launchpad, and I'm confident that after review and possibly some tweaks, it will be integrated with Drizzle.



Documentation and examples for using this new module are in upcoming posts.

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:11:"Mark Atwood";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:47;a:6:{s:4:"data";s:48:"
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:49:"PHP: Client side caching for all MySQL extensions";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:32:"http://blog.ulf-wendel.de/?p=286";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:32:"http://blog.ulf-wendel.de/?p=286";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:17170:"
The first public mysqlnd plugin adds client side query result caching to all MySQL extensions of PHP (ext/mysql, ext/mysqli, PDO_MySQL).  The cache is written in C. It does not change any of the PHP MySQL APIs and works with any PHP application using MySQL.  Query results are stored on the client. Cached data can be stored in main memory, APC, Memcache, SQLite (theoretically - via SQLite wrapper - BerkeleyDB). The default invalidation strategy is TTL (Time to live): cache entries are valid for n-seconds. The TTL can be set per query or globally for all queries.  User-defined storage handler can implement any  invalidation method. Some built-in storage handler offer a special slam defense mode which you may know from Memcache or APC. Statistics help you to identify cache candidates and to measure cache efficiency.  The plugin is available in source (PHP License)

	
	Built-in query caching for all PHP MySQL extensions/APIs



View more presentations on the MySQL native driver for PHP (mysqlnd) and its C plugin API.

	
	
Download :
	
	PDF - *.pdf, 2.3MB, 62 pages
	OpenOffice presentation - *.odt,  1.8MB, 62 slides
	
	C performance, almost transparent,  horizontal scale out
	
The mysqlnd query result plugin is written in C like every other mysqlnd plugin today. It plugs into the MySQL native driver for PHP (mysqlnd) and adds query caching functionality to it.  C gives you the best possible performance - better than any PHP based cache. The plugin is also a PHP extension. Adding a new PHP extension to an existing PHP deployment infrastructure should be easy going technically. Because it is a PHP extension and part of the database library, it gives many users, who do not compile PHP themselves, an "out-of-the box" experience. For those users caching will be just "built-in".

	
If you are new to mysqlnd and mysqlnd plugins, please note the background information given at the end of the article.

	
	
	Any PHP MySQL application
	
	
	|
	
	
	ext/mysql, ext/mysqli, PDO_MySQL
	
	
	|
	
	
	MySQL native driver for PHP (mysqlnd)
	
	
	mysqlnd query result cache plugin
	
	
	Hit
	 
	Miss
	
	
	Cache: main memory, Memcached, APC, SQLite
	 
	MySQL Server
	
	
	
The cache plugin operates on a new level in between the MySQL Server and the existing PHP MySQL APIs. Therefore it does not change any of the existing APIs.  It works with every PHP MySQL application including existing ones. This is comparable to a proxy cache.

	
	
Client based caches move contents closer to the client. This saves network traffic, makes data accesses faster and lowers the load on the MySQL Server. The MySQL Server and in particular its built-in MySQL Query Cache (don’t confuse this with the plugin!) can easily become a bottle-neck. Central resources are hard to scale.  Most of the time you have no other choice but using more powerful hardware. This approach has its limits. It is much easier to scale by client. The mysqlnd query result cache plugin allows you to scale by client. If you hit a load limit, you add a new machine. No worries on overloading the MySQL Query Cache.

	Applications control what gets cached
	
The query cache plugin has two modes of operation. It can either cache every query or only those queries marked for caching with a special SQL comment. The latter is the recommended mode. Applications control what gets cached. SQL comments (SQL hints) control the cache. The SQL hints must be placed at the very beginning of query.

	

	
$mysqli = new mysqli("localhost", "user", "password", "database");
$mysql->query("⁄*qc=on*⁄SELECT something FROM slowtable");
	
$pdo = new PDO("mysql:host=localhost;dbname=database", "user", "password");
$stmt = $pdo->execute("⁄*qc=on*⁄SELECT something FROM slowtable");
	
$mysql = mysql_connect("localhost", "user", "password", true);
mysql_select_db("database", $mysql);
mysql_query("⁄*qc=on*⁄SELECT something FROM slowtable");

	

	
If you run the cache plugin in the unrecommended mode of caching all queries by default you can disable automatic caching for a particular query by preceding it with the SQL hint “/*qc=off*/”. If you use the recommended mode of caching only those queries marked for caching you have to use the SQL hint “/*qc=on*/” to enable caching. Queries will be cached for the number of seconds specified with the PHP configuration setting (php.ini) mysqlnd_qc.ttl. The default TTL can be overwritten on a per query basis using the SQL hint “/*qc_ttl=n*/”.

	 That is it - you have mastered the foundations of using the plugin in your application. No worries dear experts, the standard TTL invalidation strategy can be replaced by user-defined ones.

	Proven standard solutions for storing data
	
The cache plugin can store cache entries in many different media. For storing data in the main memory of the PHP process the cache uses its own storage handler based on hashes from by the PHP runtime environment. For storing data in shared memory, for using memory-mapped I/O, for using files and for using a server deamon which uses main memory the plugin reuses proven open standard solutions. You may be using some of them already, the cache plugin may fit nicely into your setup.

	
	
	Handler
	Medium
	Location
	
	
	Default
	Main memory of the PHP process (Hashes)
	local
	
	
	Memcached
	Deamon using main memory
	local or remote
	
	
	APC
	Shared memory, memory-mapped I/O
	local
	
	
	SQLite
	Main memory of the PHP process or file
	local
	
	
	Userdefined
	No limits - whatever you can envision
	local or remote
	
	
	
The storage handler differ from each other. Some are known to scale extremly well, others are not. Some can handle high concurrent loads, some can not. Some can share their data between multiple PHP processes or even multiple machines, others may not be able to reuse a cache entry for more than the duration of a web request. On top of the table you see the combinations which are bad at reusing cache entries. At the end of the table are those combinations which give you the best possible reuse.

	
	
	Handler
	Deployment
	Cache entries are reusable/shared…
	
	
	Default
	CGI
	within single web request
	
	
	SQLite
	CGI and :memory:
	within single web request
	
	
	Default
	FastCGI/Prefork/Threaded
	single process, multiple requests
	
	
	SQLite
	FastCGI/Prefork/Threaded and :memory:
	single process, multiple requests
	
	
	APC
	Shared memory
	multiple processes, single machine
	
	
	APC
	Memory-mapped I/O
	multiple processes, single machine (yes, NFS but…)
	
	
	SQLite
	file - you don’t want this!
	multiple processes, single maschine
	
	
	Memcache
	local server
	multiple processes, single maschine
	
	
	Memcache
	remote server
	multiple maschines
	
	
	
You have the choice of picking the combination that fits your PHP MySQL application best.

	
	
	Scope of a cache entry
	Handler
	
	
	single request
	Default (CGI), SQLite (CGI and :memory:), user-defined
	
	
	single process (mutiple requests)
	Default (FastCGI/prefork/threaded), SQLite (FastCGI/prefork/threaded and :memory:), user-defined
	
	
	single machine (mutiple processes)
	APC (Shared memory, memory-mapped I/O), Memcached [, SQLite (file)], user-defined
	
	
	multiple machines
	Memcached, user-defined
	
	
	The unlimited offering: user-defined storage handler
	
Internally the mysqlnd cache plugin consists of a core and the various storage handler. The core has no more logic in it than absolutely necessary.  Storage handler are given as much freedom and as many responsibilities as possible. Storage handler do:
	
	control which query gets cached
	control where to store data
	implement the invalidation strategy
	maintain cache statistics
	
	PHP developers can write user-defined storage handler in PHP to break out of the limits of the built-in handlers. For example, a user-defined storage handler could maintain a dependency graph between cache entries and automatically invalidate all dependent cache entries if underlying data changes.  This may lower the risk of serving stale data, which is possible if using the standard TTL invalidation strategy. You can  support any invalidation strategy - even those we have not been able to think of.

	
User-defined storage handler could also eliminate the need of having to use SQL hints. You could teach your storage handler what queries it shall cache regardless if they start with a SQL hint or not. No need to touch the code of an existing application, no need to change any SQL query. Or say, you want to cache all queries which have a run-time of more than 0.1s - no problem, hack a storage handler.

	
User defined storage handler can be written by:
	
	providing procedural callback functions
	implementing an interface and registering the object
	subclassing and specializing the built-in default storage handler
	
	 Procedural - checked, object oriented - checked, extending build in storage handler - checked… choose one.

	Slam defense
	
Basic caching for a simple application is super easy: use the plugin, add some SQL hints, let the driver do the work and stop bothering.  But caching for high loads is an expert topic. It requires careful planning.

	All the major caches, including APC and Memcached, have learned a lesson on overloading and reusing cache entries: what if a very popular cache entry expires?

	
	
	Client 1
	 
	Client 2
	Client 3
	Client 3
	Client ..
	Client n
	
	
	|
	 
	|
	
	
	|
	 
	Cache
	
	
	|
	 
	 
	
	
	MySQL Server
	
	
	Load
	
	
	 
	
	
	 
	
	
	 
	 
	 
	 
	 
	 
	 
	
	
	 All of a sudden many clients are no longer served by the cache. They get a cache miss and start quering the data source. A sudden load peak on the data source is the consequence. The MySQL Server, as an example of a data source, will become very busy. Because it start to be overloaded it takes longer and longer to compute answers for all the requests. Over the time more and more cache entries expire because the data source is still busy calculating a reply for the first client that got a cache miss and the cache has not been refreshed yet. The MySQL Server is going to die because you of the cache!

	
	
	Client 1
	 
	Client 2
	Client 3
	Client 3
	Client ..
	Client n
	
	
	|
	 
	|
	
	
	|
	 
	Cache Miss
	
	
	|
	 
	|
	
	
	MySQL Server
	
	
	Load
	
	
	 
	*tilt*
	
	
	 
	 
	 
	
	
	 
	 
	
	
	
To prevent slamming the MySQL Server the plugin can continue to serve the expired cache entry until they get refreshed. The first client that hits an expired cache entry gets a cache miss. It will query the data source and update the cache entry. But all subsequent clients accessing the same cache entry will get a cache hit until the first client has updated the cache entry. The subsequent clients are given a cache hit although the cache entry is experired.
	
	
	Client 1
	 
	Client 2
	Client 3
	Client 3
	Client ..
	Client n
	
	
	|
	 
	|
	
	
	|
	 
	|
	Cache (expired but slam defense)
	
	
	|
	 
	|
	 
	
	
	MySQL Server
	
	
	Load
	
	
	 
	
	
	 
	 
	 
	
	
	 
	 
	 
	
	
	
All the big boys have this feature. We have it too. We call it slam defense because this is how APC, a PHP specific solution, calls it.

	
It is not possible for the cache plugin to track if the first client which got a cache miss and is supposed to update an expired cache entry has managed to update the entry.  We do not know if the client died or not. To prevent an expired cache entry from being in slam defense mode and being served forever we use a TTL for slam defense. If slam defense is turned on an expired cache entry is served for another n-seconds according to the mysqlnd_qc.slam_defense_ttl PHP configuration setting (php.ini) it automatically expired. In the worst case slamming may still happen. But the cache tried its best to prevent it.

	A PHP based solution cannot compete
	
Storage handler are given raw wire data to cache. All existing PHP based query result caching solutions cache the PHP user space result of a query. They cache the PHP array or PHP object which holds the rows of a query result. It is almost impossible to persist the C structures which represent a PHP variable. PHP variables cannot survive the end of PHP process. PHP based caches have to serialize variables before adding them to a cache and have to unserialize them when fetching from the cache. The mysqlnd query cache plugin skips the serialization task.

	
The query cache plugin hooks the mysqlnd calls for reading and sending raw wire data. It records the wire data send from MySQL to PHP. In case of a cache hit it replays the raw wire data. Upon replay the raw wire data needs to be decoded by mysqlnd. This is a very simple and lean solution: no serialization needed.  It is impossible for a PHP based solution to do the same. The PHP based solution cannot manipulate the mysqlnd network functions. This is only possible with the mysqlnd C plugin API.

	Limitations
	
After all the good news a bad one. The cache plugin works only with buffered queries. It cannot cache unbuffered ones. However, the most popular PHP MySQL API calls stand for buffered queries:
	
	ext/mysqli:

	mysqli_query()
	mysqli_query() + mysqli_store_result()
	
	
	PDO_MySQL:

	PDO::query(), PDO::exec(), PDO::prepare() if PDO::ATTR_EMULATE_PREPARES = 1 (which is the default setting!)
	

	ext/mysql:

	mysql_query()
	
	
	
	
The cache plugin cannot cache any of the following unbuffered queries:
	
	ext/mysqli:

	mysqli_real_query() + mysqli_use_result()
	mysqli_query() + MYSQLI_ASYNC
	mysqli_stmt_*()
	
	
	PDO_MySQL:

	PDO::query(), PDO::exec(), PDO::prepare() if PDO::ATTR_EMULATE_PREPARES = 0
	

	ext/mysql:

	mysql_unbuffered_query()
	
	
	
	Download and further readings
	
Please find the source code at the project’s MySQL Forge wiki page at http://forge.mysql.com/wiki/MySQLnd_Query_Cache_Plugin_for_PHP.  The cache plugin has been proposed to PECL. If the PHP PECL community wants it, you may be able to download it from PECL in the future. If not, it is probably going to Launchpad. In any case, the download location will change. The wiki page will point you to whatever may the current location be.

	
We release the software as a prototype to stress out that it is not an official MySQL product for which you can buy commercial support. Technically it is on beta level. It has been successfully build on more than 25 platforms and its tests cover well above 85% of the source code.

	
Documentation on run time configuration and user space functions of the query cache plugin is available on the wiki page.

	Comments are welcome, credits to Andrey
	
As usual, we are happy to answer comments and questions. The is the first public and downloadable plugin. We are eager to see how it will be received and what other plugins you want from us. And, as usual, credits to Andrey who did most of the query cache coding.

	Happy hacking!
	
	Background: mysqlnd plugins
	
The MySQL native driver for PHP (mysqlnd) is a drop-in replacement for the MySQL Client Library (AKA libmysql, libmysqlclient). Mysqlnd is a C library like the MySQL Client Library. It can be used to connect to MySQL. The mysqlnd library is part of PHP since PHP 5.3.  All PHP MySQL extensions (ext/mysql, ext/mysqli, PDO_MySQL) can either make use of mysqlnd or the MySQL Client Library. Which one gets used is determined at compile time.

	
	
	
	Drupal, phpMyFAQ, phpMyAdmin, Oxid, …
	
	
	|
	
	
	ext/mysql, ext/mysqli, ext/PDO_MYSQL
	
	
	Mysqlnd
	
	
	Mysqlnd plugin
	
	
	Load Balancing
	Monitoring
	Performance: Cache
	
	
	|
	
	
	MySQL Server
	
	
	
A mysqlnd plugin can add new functionality to mysqlnd. Plugins are written in C. Because plugins and mysqlnd operate "under the hood" of PHP they can be understood as a transparent proxy from a user perspective. A mysqlnd plugin does not change any user space APIs. Therefore it is transparent from a PHP application point of view. Plugins can be often be used with existing applications without having to change anything in the application. That is because plugins operate on a new layer in beneath the PHP application and above the MySQL server.

	
Plugins such as the mysqlnd query result plugin make use of the mysqlnd plugin API. Because mysqlnd is written in C the plugin API is C based.  The foundations of the plugin API have been layed in PHP 5.3.0. Back then the lead developer of mysqlnd, Andrey Hristov, did not plan to create a plugin API.  It sort of just happened…. Today’s mysqlnd plugin API exists since PHP 5.3.2. C developers can find a thorough  description of how to create a plugin here.  Further information is also given in the blog posting mysqlnd plugins: alternative to MySQL Proxy ?! or in the slides show from below.

	
	The PHP mysqlnd plugin talk - plugins an alternative to MySQL Proxy



View more presentations from Ulf Wendel.

	";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Thu, 24 Jun 2010 11:06:06 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:2:{i:0;a:5:{s:4:"data";s:21:"PlanetMySQL (english)";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:19:"PlanetPHP (english)";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:28711:"

The first public mysqlnd plugin adds client side query result caching to all MySQL extensions of PHP (ext/mysql, ext/mysqli, PDO_MySQL).  The cache is written in C. It does not change any of the PHP MySQL APIs and works with any PHP application using MySQL.  Query results are stored on the client. Cached data can be stored in main memory, APC, Memcache, SQLite (theoretically - via SQLite wrapper - BerkeleyDB). The default invalidation strategy is TTL (Time to live): cache entries are valid for n-seconds. The TTL can be set per query or globally for all queries.  User-defined storage handler can implement any  invalidation method. Some built-in storage handler offer a special slam defense mode which you may know from Memcache or APC. Statistics help you to identify cache candidates and to measure cache efficiency.  The plugin is available in source (PHP License)


	

	
Built-in query caching for all PHP MySQL extensions/APIs






View more presentations on the MySQL native driver for PHP (mysqlnd) and its C plugin API.



	

	

Download :

	
	
C performance, almost transparent,  horizontal scale out

	

The mysqlnd query result plugin is written in C like every other mysqlnd plugin today. It plugs into the MySQL native driver for PHP (mysqlnd) and adds query caching functionality to it.  C gives you the best possible performance - better than any PHP based cache. The plugin is also a PHP extension. Adding a new PHP extension to an existing PHP deployment infrastructure should be easy going technically. Because it is a PHP extension and part of the database library, it gives many users, who do not compile PHP themselves, an "out-of-the box" experience. For those users caching will be just "built-in".


	

If you are new to mysqlnd and mysqlnd plugins, please note the background information given at the end of the article.


	

	

	Any PHP MySQL application
	

	

	|
	

	

	ext/mysql, ext/mysqli, PDO_MySQL
	

	

	|
	

	

	MySQL native driver for PHP (mysqlnd)
	

	

	mysqlnd query result cache plugin
	

	

	Hit
	 
	Miss
	

	

	Cache:
 main memory, Memcached, APC, SQLite		
	 
	MySQL Server
	

	

	

The cache plugin operates on a new level in between the MySQL Server and the existing PHP MySQL APIs. Therefore it does not change any of the existing APIs.  It works with every PHP MySQL application including existing ones. This is comparable to a proxy cache.


	

	

Client based caches move contents closer to the client. This saves network traffic, makes data accesses faster and lowers the load on the MySQL Server. The MySQL Server and in particular its built-in MySQL Query Cache (don’t confuse this with the plugin!) can easily become a bottle-neck. Central resources are hard to scale.  Most of the time you have no other choice but using more powerful hardware. This approach has its limits. It is much easier to scale by client. The mysqlnd query result cache plugin allows you to scale by client. If you hit a load limit, you add a new machine. No worries on overloading the MySQL Query Cache.


	
Applications control what gets cached

	

The query cache plugin has two modes of operation. It can either cache every query or only those queries marked for caching with a special SQL comment. The latter is the recommended mode. Applications control what gets cached. SQL comments (SQL hints) control the cache. The SQL hints must be placed at the very beginning of query.


	



	
$mysqli = new mysqli("localhost", "user", "password", "database");
$mysql->query("⁄*qc=on*⁄SELECT something FROM slowtable");
	
$pdo = new PDO("mysql:host=localhost;dbname=database", "user", "password");
$stmt = $pdo->execute("⁄*qc=on*⁄SELECT something FROM slowtable");
	
$mysql = mysql_connect("localhost", "user", "password", true);
mysql_select_db("database", $mysql);
mysql_query("⁄*qc=on*⁄SELECT something FROM slowtable");


	



	

If you run the cache plugin in the unrecommended mode of caching all queries by default you can disable automatic caching for a particular query by preceding it with the SQL hint “/*qc=off*/”. If you use the recommended mode of caching only those queries marked for caching you have to use the SQL hint “/*qc=on*/” to enable caching. Queries will be cached for the number of seconds specified with the PHP configuration setting (php.ini) mysqlnd_qc.ttl. The default TTL can be overwritten on a per query basis using the SQL hint “/*qc_ttl=n*/”.


	
 That is it - you have mastered the foundations of using the plugin in your application. No worries dear experts, the standard TTL invalidation strategy can be replaced by user-defined ones.


	
Proven standard solutions for storing data

	

The cache plugin can store cache entries in many different media. For storing data in the main memory of the PHP process the cache uses its own storage handler based on hashes from by the PHP runtime environment. For storing data in shared memory, for using memory-mapped I/O, for using files and for using a server deamon which uses main memory the plugin reuses proven open standard solutions. You may be using some of them already, the cache plugin may fit nicely into your setup.


	

	

	Handler
	Medium
	Location
	

	

	Default
	Main memory of the PHP process (Hashes)
	local
	

	

	Memcached
	Deamon using main memory
	local or remote
	

	

	APC
	Shared memory, memory-mapped I/O
	local
	

	

	SQLite
	Main memory of the PHP process or file
	local
	

	

	Userdefined
	No limits - whatever you can envision
	local or remote
	

	

	

The storage handler differ from each other. Some are known to scale extremly well, others are not. Some can handle high concurrent loads, some can not. Some can share their data between multiple PHP processes or even multiple machines, others may not be able to reuse a cache entry for more than the duration of a web request. On top of the table you see the combinations which are bad at reusing cache entries. At the end of the table are those combinations which give you the best possible reuse.


	

	

	Handler
	Deployment
	Cache entries are reusable/shared…
	

	

	Default
	CGI
	within single web request
	

	

	SQLite
	CGI and :memory:
	within single web request
	

	

	Default
	FastCGI/Prefork/Threaded
	single process, multiple requests
	

	

	SQLite
	FastCGI/Prefork/Threaded and :memory:
	single process, multiple requests
	

	

	APC
	Shared memory
	multiple processes, single machine
	

	

	APC
	Memory-mapped I/O
	multiple processes, single machine (yes, NFS but…)
	

	

	SQLite
	file - you don’t want this!
	multiple processes, single maschine
	

	

	Memcache
	local server
	multiple processes, single maschine
	

	

	Memcache
	remote server
	multiple maschines
	

	

	

You have the choice of picking the combination that fits your PHP MySQL application best.


	

	

	Scope of a cache entry
	Handler
	

	

	single request
	Default (CGI), SQLite (CGI and :memory:), user-defined
	

	

	single process (mutiple requests)
	Default (FastCGI/prefork/threaded), SQLite (FastCGI/prefork/threaded and :memory:), user-defined
	

	

	single machine (mutiple processes)
	APC (Shared memory, memory-mapped I/O), Memcached [, SQLite (file)], user-defined
	

	

	multiple machines
	Memcached, user-defined
	

	

	
The unlimited offering: user-defined storage handler

	

Internally the mysqlnd cache plugin consists of a core and the various storage handler. The core has no more logic in it than absolutely necessary.  Storage handler are given as much freedom and as many responsibilities as possible. Storage handler do:

	
	
PHP developers can write user-defined storage handler in PHP to break out of the limits of the built-in handlers. For example, a user-defined storage handler could maintain a dependency graph between cache entries and automatically invalidate all dependent cache entries if underlying data changes.  This may lower the risk of serving stale data, which is possible if using the standard TTL invalidation strategy. You can  support any invalidation strategy - even those we have not been able to think of.


	

User-defined storage handler could also eliminate the need of having to use SQL hints. You could teach your storage handler what queries it shall cache regardless if they start with a SQL hint or not. No need to touch the code of an existing application, no need to change any SQL query. Or say, you want to cache all queries which have a run-time of more than 0.1s - no problem, hack a storage handler.


	

User defined storage handler can be written by:

	
	
 Procedural - checked, object oriented - checked, extending build in storage handler - checked… choose one.


	
Slam defense

	

Basic caching for a simple application is super easy: use the plugin, add some SQL hints, let the driver do the work and stop bothering.  But caching for high loads is an expert topic. It requires careful planning.


	
All the major caches, including APC and Memcached, have learned a lesson on overloading and reusing cache entries: what if a very popular cache entry expires?


	

	

	Client 1
	 
	Client 2
	Client 3
	Client 3
	Client ..
	Client n
	

	

	|
	 
	|
	

	

	|
	 
	Cache
	

	

	|
	 
	 
	

	

	MySQL Server
	

	

	Load
	

	

	 
	

	

	 
	

	

	 
	 
	 
	 
	 
	 
	 
	

	

	
 All of a sudden many clients are no longer served by the cache. They get a cache miss and start quering the data source. A sudden load peak on the data source is the consequence. The MySQL Server, as an example of a data source, will become very busy. Because it start to be overloaded it takes longer and longer to compute answers for all the requests. Over the time more and more cache entries expire because the data source is still busy calculating a reply for the first client that got a cache miss and the cache has not been refreshed yet. The MySQL Server is going to die because you of the cache!


	

	

	Client 1
	 
	Client 2
	Client 3
	Client 3
	Client ..
	Client n
	

	

	|
	 
	|
	

	

	|
	 
	Cache Miss
	

	

	|
	 
	|
	

	

	MySQL Server
	

	

	Load
	

	

	 
	*tilt*
	

	

	 
	 
	 
	

	

	 
	 
	

	

	

To prevent slamming the MySQL Server the plugin can continue to serve the expired cache entry until they get refreshed. The first client that hits an expired cache entry gets a cache miss. It will query the data source and update the cache entry. But all subsequent clients accessing the same cache entry will get a cache hit until the first client has updated the cache entry. The subsequent clients are given a cache hit although the cache entry is experired.

	

	

	Client 1
	 
	Client 2
	Client 3
	Client 3
	Client ..
	Client n
	

	

	|
	 
	|
	

	

	|
	 
	|
	Cache (expired but slam defense)
	

	

	|
	 
	|
	 
	

	

	MySQL Server
	

	

	Load
	

	

	 
	

	

	 
	 
	 
	

	

	 
	 
	 
	

	

	

All the big boys have this feature. We have it too. We call it slam defense because this is how APC, a PHP specific solution, calls it.


	

It is not possible for the cache plugin to track if the first client which got a cache miss and is supposed to update an expired cache entry has managed to update the entry.  We do not know if the client died or not. To prevent an expired cache entry from being in slam defense mode and being served forever we use a TTL for slam defense. If slam defense is turned on an expired cache entry is served for another n-seconds according to the mysqlnd_qc.slam_defense_ttl PHP configuration setting (php.ini) it automatically expired. In the worst case slamming may still happen. But the cache tried its best to prevent it.


	
A PHP based solution cannot compete

	

Storage handler are given raw wire data to cache. All existing PHP based query result caching solutions cache the PHP user space result of a query. They cache the PHP array or PHP object which holds the rows of a query result. It is almost impossible to persist the C structures which represent a PHP variable. PHP variables cannot survive the end of PHP process. PHP based caches have to serialize variables before adding them to a cache and have to unserialize them when fetching from the cache. The mysqlnd query cache plugin skips the serialization task.


	

The query cache plugin hooks the mysqlnd calls for reading and sending raw wire data. It records the wire data send from MySQL to PHP. In case of a cache hit it replays the raw wire data. Upon replay the raw wire data needs to be decoded by mysqlnd. This is a very simple and lean solution: no serialization needed.  It is impossible for a PHP based solution to do the same. The PHP based solution cannot manipulate the mysqlnd network functions. This is only possible with the mysqlnd C plugin API.


	
Limitations

	

After all the good news a bad one. The cache plugin works only with buffered queries. It cannot cache unbuffered ones. However, the most popular PHP MySQL API calls stand for buffered queries:

	
	

The cache plugin cannot cache any of the following unbuffered queries:

	
	
Download and further readings

	

Please find the source code at the project’s MySQL Forge wiki page at http://forge.mysql.com/wiki/MySQLnd_Query_Cache_Plugin_for_PHP.  The cache plugin has been proposed to PECL. If the PHP PECL community wants it, you may be able to download it from PECL in the future. If not, it is probably going to Launchpad. In any case, the download location will change. The wiki page will point you to whatever may the current location be.


	

We release the software as a prototype to stress out that it is not an official MySQL product for which you can buy commercial support. Technically it is on beta level. It has been successfully build on more than 25 platforms and its tests cover well above 85% of the source code.


	

Documentation on run time configuration and user space functions of the query cache plugin is available on the wiki page.


	
Comments are welcome, credits to Andrey

	

As usual, we are happy to answer comments and questions. The is the first public and downloadable plugin. We are eager to see how it will be received and what other plugins you want from us. And, as usual, credits to Andrey who did most of the query cache coding.


	
Happy hacking!

	

	
Background: mysqlnd plugins

	

The MySQL native driver for PHP (mysqlnd) is a drop-in replacement for the MySQL Client Library (AKA libmysql, libmysqlclient). Mysqlnd is a C library like the MySQL Client Library. It can be used to connect to MySQL. The mysqlnd library is part of PHP since PHP 5.3.  All PHP MySQL extensions (ext/mysql, ext/mysqli, PDO_MySQL) can either make use of mysqlnd or the MySQL Client Library. Which one gets used is determined at compile time.


	

	

	

	Drupal, phpMyFAQ, phpMyAdmin, Oxid, …
	

	

	|
	

	

	ext/mysql, ext/mysqli, ext/PDO_MYSQL
	

	

	Mysqlnd
	

	

	Mysqlnd plugin
	

	

	Load Balancing
	Monitoring
	Performance: Cache
	

	

	|
	

	

	MySQL Server
	

	

	

A mysqlnd plugin can add new functionality to mysqlnd. Plugins are written in C. Because plugins and mysqlnd operate "under the hood" of PHP they can be understood as a transparent proxy from a user perspective. A mysqlnd plugin does not change any user space APIs. Therefore it is transparent from a PHP application point of view. Plugins can be often be used with existing applications without having to change anything in the application. That is because plugins operate on a new layer in beneath the PHP application and above the MySQL server.


	

Plugins such as the mysqlnd query result plugin make use of the mysqlnd plugin API. Because mysqlnd is written in C the plugin API is C based.  The foundations of the plugin API have been layed in PHP 5.3.0. Back then the lead developer of mysqlnd, Andrey Hristov, did not plan to create a plugin API.  It sort of just happened…. Today’s mysqlnd plugin API exists since PHP 5.3.2. C developers can find a thorough  description of how to create a plugin here.  Further information is also given in the blog posting mysqlnd plugins: alternative to MySQL Proxy ?! or in the slides show from below.


	

	
The PHP mysqlnd plugin talk - plugins an alternative to MySQL Proxy






View more presentations from Ulf Wendel.



	

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:10:"Ulf Wendel";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:48;a:6:{s:4:"data";s:88:"
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:6:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:46:"Nginx-Fu: X-Accel-Redirect From Remote Servers";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:25:"http://kovyrin.net/?p=462";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:59:"http://feedproxy.google.com/~r/Homo-Adminus/~3/cuG-f1auDlo/";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:3502:"We use nginx and its features a lot in Scribd. Many times in the last year we needed some pretty interesting, but not supported feature – we wanted nginx X-Accel-Redirect functionality to work with remote URLs. Out of the box nginx supports this functionality for local URIs only. In this short post I want to explain how did we make nginx serve remote content via X-Accel-Redirect.

First of all, here is why you may need this feature. Let’s imagine you have a file storage on Amazon S3 where you store tons of content. And you have an application where you have some content downloading functionality that you want to be available for logged-in/paying/premium users and/or you want to keep track of downloads your users perform on your site. If your content was on your web server, you could have used simple controlled downloads functionality built-in to nginx out of the box. But the problem is that your content is remote.
Here is what we do to solve this problem.
First, we create a special location on our nginx server. This location will be used as a proxy for all our accelerated file downloads:
1234567891011121314151617181920212223242526272829303132333435# Proxy download 
location ~* ^/internal_redirect/(.*?)/(.*) {
  # Do not allow people to mess with this location directly
  # Only internal redirects are allowed
  internal;

  # Location-specific logging
  access_log logs/internal_redirect.access.log main;
  error_log logs/internal_redirect.error.log warn;

  # Extract download url from the request
  set $download_uri $2;
  set $download_host $1;

  # Compose download url
  set $download_url http://$download_host/$download_uri;

  # Set download request headers
  proxy_set_header Host $download_host;
  proxy_set_header Authorization '';

  # The next two lines could be used if your storage 
  # backend does not support Content-Disposition 
  # headers used to specify file name browsers use 
  # when save content to the disk
  proxy_hide_header Content-Disposition;
  add_header Content-Disposition 'attachment; filename="$args"';

  # Do not touch local disks when proxying 
  # content to clients
  proxy_max_temp_file_size 0;

  # Download the file and send it to client
  proxy_pass $download_url;
}
After adding this location to our nginx config we could start sending responses with headers like the following:
1234567# This header will ask nginx to download a file 
# from http://some.site.com/secret/url.ext and send it to user
X-Accel-Redirect: /internal_redirect/some.site.com/secret/url.ext

# This header will ask nginx to download a file 
# from http://blah.com/secret/url and send it to user as cool.pdf
X-Accel-Redirect: /internal_redirect/blah.com/secret/url?cool.pdf
Here is an example code you could use in a Rails application to use our internal redirect location:
12345678910def x_accel_url(url, file_name = nil)
  uri = "/internal_redirect/#{url.gsub('http://', '')}"
  uri << "?#{file_name}" if file_name
  return uri
end

def download
  headers['X-Accel-Redirect'] = x_accel_url(some_secret_url, pretty_name)
  render :nothing => true
end
As you can see, nginx is really powerful tool and when you turn your creativity on you can make it even more powerful. Stay tuned for more Nginx-Fu posts.



  
";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Thu, 24 Jun 2010 03:45:52 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:8:"category";a:10:{i:0;a:5:{s:4:"data";s:10:"Admin-tips";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:1;a:5:{s:4:"data";s:11:"Development";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:2;a:5:{s:4:"data";s:8:"Networks";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:3;a:5:{s:4:"data";s:6:"amazon";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:4;a:5:{s:4:"data";s:8:"download";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:5;a:5:{s:4:"data";s:5:"Nginx";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:6;a:5:{s:4:"data";s:8:"nginx-fu";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:7;a:5:{s:4:"data";s:8:"redirect";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:8;a:5:{s:4:"data";s:2:"S3";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}i:9;a:5:{s:4:"data";s:16:"x-accel-redirect";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:6926:"
We use nginx and its features a lot in Scribd. Many times in the last year we needed some pretty interesting, but not supported feature – we wanted nginx X-Accel-Redirect functionality to work with remote URLs. Out of the box nginx supports this functionality for local URIs only. In this short post I want to explain how did we make nginx serve remote content via X-Accel-Redirect.




First of all, here is why you may need this feature. Let’s imagine you have a file storage on Amazon S3 where you store tons of content. And you have an application where you have some content downloading functionality that you want to be available for logged-in/paying/premium users and/or you want to keep track of downloads your users perform on your site. If your content was on your web server, you could have used simple controlled downloads functionality built-in to nginx out of the box. But the problem is that your content is remote.


Here is what we do to solve this problem.


First, we create a special location on our nginx server. This location will be used as a proxy for all our accelerated file downloads:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# Proxy download 

location ~* ^/internal_redirect/(.*?)/(.*) {

  # Do not allow people to mess with this location directly

  # Only internal redirects are allowed

  internal;



  # Location-specific logging

  access_log logs/internal_redirect.access.log main;

  error_log logs/internal_redirect.error.log warn;



  # Extract download url from the request

  set $download_uri $2;

  set $download_host $1;



  # Compose download url

  set $download_url http://$download_host/$download_uri;



  # Set download request headers

  proxy_set_header Host $download_host;

  proxy_set_header Authorization '';



  # The next two lines could be used if your storage 

  # backend does not support Content-Disposition 

  # headers used to specify file name browsers use 

  # when save content to the disk

  proxy_hide_header Content-Disposition;

  add_header Content-Disposition 'attachment; filename="$args"';



  # Do not touch local disks when proxying 

  # content to clients

  proxy_max_temp_file_size 0;



  # Download the file and send it to client

  proxy_pass $download_url;

}


After adding this location to our nginx config we could start sending responses with headers like the following:


1
2
3
4
5
6
7
# This header will ask nginx to download a file 

# from http://some.site.com/secret/url.ext and send it to user

X-Accel-Redirect: /internal_redirect/some.site.com/secret/url.ext



# This header will ask nginx to download a file 

# from http://blah.com/secret/url and send it to user as cool.pdf

X-Accel-Redirect: /internal_redirect/blah.com/secret/url?cool.pdf


Here is an example code you could use in a Rails application to use our internal redirect location:


1
2
3
4
5
6
7
8
9
10
def x_accel_url(url, file_name = nil)

  uri = "/internal_redirect/#{url.gsub('http://', '')}"

  uri << "?#{file_name}" if file_name

  return uri

end



def download

  headers['X-Accel-Redirect'] = x_accel_url(some_secret_url, pretty_name)

  render :nothing => true

end


As you can see, nginx is really powerful tool and when you turn your creativity on you can make it even more powerful. Stay tuned for more Nginx-Fu posts.







  


PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:14:"Alexey Kovyrin";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}i:49;a:6:{s:4:"data";s:38:"
    
    
    
    
    
    
    
  ";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";s:5:"child";a:3:{s:0:"";a:5:{s:5:"title";a:1:{i:0;a:5:{s:4:"data";s:23:"Hadoop for MySQL people";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"guid";a:1:{i:0;a:5:{s:4:"data";s:61:"tag:everythingmysql.ning.com,2010-06-24:3993569:BlogPost:1816";s:7:"attribs";a:1:{s:0:"";a:1:{s:11:"isPermaLink";s:5:"false";}}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:4:"link";a:1:{i:0;a:5:{s:4:"data";s:67:"http://everythingmysql.ning.com/xn/detail/3993569%3ABlogPost%3A1816";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:11:"description";a:1:{i:0;a:5:{s:4:"data";s:1546:"There's a lot of buzz lately about Hadoop. If you're completely new to Hadoop, I recommend the free videos from Cloudera (http://www.cloudera.com/resources/?type=Training). If you have a vague idea and want to play around, it's easy!
First, download Cloudera's training VM which has a small Hadoop cluster already installed and running:
http://www.cloudera.com/developers/downloads/virtual-machine/

Second, you need to put some data into Hadoop. Fortunately for database folks, there's a tool to import data into Hadoop from MySQL called "Sqoop". It's already installed on the VM and there are instructions for using Sqoop to import some MySQL tables into Hadoop (see Desktop/instructions/exercises/SqoopExercise.html inside the VM). FYI, it's not uncommon to "Sqoop" data into Hadoop, do analysis and transformations, and then use Sqoop to export the data back to MySQL.

Now you're ready to do analysis of your data using Hadoop's powerful MapReduce. Except that MapReduce requires coding (Java, Python, PHP, etc) and an understanding of the functional programming model that is MapReduce. For an easier entry into Hadoop, try Hive. Hive is a data warehousing system for Hadoop. It offers a language (HiveQL) that feels just like SQL. Examples:

$ hive
hive> SHOW TABLES;
hive> SELECT * FROM <table> LIMIT 10;

Hive supports most of the SQL queries you are used to. For example JOIN, LEFT OUTER JOIN, RIGHT OUTER JOIN, GROUP BY, ORDER BY, aggregate functions, etc. The best part is that Hive can scale to analyze petabytes of data!";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}s:7:"pubDate";a:1:{i:0;a:5:{s:4:"data";s:31:"Thu, 24 Jun 2010 01:04:09 +0000";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:40:"http://purl.org/rss/1.0/modules/content/";a:1:{s:7:"encoded";a:1:{i:0;a:5:{s:4:"data";s:2066:"There's a lot of buzz lately about Hadoop. If you're completely new to Hadoop, I recommend the free videos from Cloudera (http://www.cloudera.com/resources/?type=Training). If you have a vague idea and want to play around, it's easy!



First, download Cloudera's training VM which has a small Hadoop cluster already installed and running:


http://www.cloudera.com/developers/downloads/virtual-machine/





Second, you need to put some data into Hadoop. Fortunately for database folks, there's a tool to import data into Hadoop from MySQL called "Sqoop". It's already installed on the VM and there are instructions for using Sqoop to import some MySQL tables into Hadoop (see Desktop/instructions/exercises/SqoopExercise.html inside the VM). FYI, it's not uncommon to "Sqoop" data into Hadoop, do analysis and transformations, and then use Sqoop to export the data back to MySQL.





Now you're ready to do analysis of your data using Hadoop's powerful MapReduce. Except that MapReduce requires coding (Java, Python, PHP, etc) and an understanding of the functional programming model that is MapReduce. For an easier entry into Hadoop, try Hive. Hive is a data warehousing system for Hadoop. It offers a language (HiveQL) that feels just like SQL. Examples:





$ hive


hive> SHOW TABLES;


hive> SELECT * FROM <table> LIMIT 10;





Hive supports most of the SQL queries you are used to. For example JOIN, LEFT OUTER JOIN, RIGHT OUTER JOIN, GROUP BY, ORDER BY, aggregate functions, etc. The best part is that Hive can scale to analyze petabytes of data!

PlanetMySQL Voting:
	 Vote UP /
	 Vote DOWN";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}s:32:"http://purl.org/dc/elements/1.1/";a:1:{s:7:"creator";a:1:{i:0;a:5:{s:4:"data";s:15:"Chris Schneider";s:7:"attribs";a:0:{}s:8:"xml_base";s:0:"";s:17:"xml_base_explicit";b:0;s:8:"xml_lang";s:0:"";}}}}}}}}}}}}}}}}s:4:"type";i:128;s:7:"headers";a:7:{s:4:"date";s:29:"Wed, 30 Jun 2010 09:31:08 GMT";s:6:"server";s:22:"Apache/2.2.15 (Fedora)";s:13:"last-modified";s:29:"Wed, 30 Jun 2010 09:16:18 GMT";s:13:"accept-ranges";s:5:"bytes";s:14:"content-length";s:6:"435060";s:10:"connection";s:5:"close";s:12:"content-type";s:8:"text/xml";}s:5:"build";s:14:"20090627192103";}