<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>My IT-blog &#187; mysql</title>
	<atom:link href="http://dev.eek.be/tag/mysql/feed/" rel="self" type="application/rss+xml" />
	<link>http://dev.eek.be</link>
	<description>IT knowledge exchange</description>
	<lastBuildDate>Sun, 18 Dec 2011 10:37:02 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3</generator>
		<item>
		<title>Split and import a very large database with PHP</title>
		<link>http://dev.eek.be/2011/04/split-and-import-a-very-large-database-with-php/</link>
		<comments>http://dev.eek.be/2011/04/split-and-import-a-very-large-database-with-php/#comments</comments>
		<pubDate>Mon, 11 Apr 2011 07:03:59 +0000</pubDate>
		<dc:creator>Wim</dc:creator>
				<category><![CDATA[linux]]></category>
		<category><![CDATA[mysql]]></category>
		<category><![CDATA[PHP]]></category>
		<category><![CDATA[servers]]></category>
		<category><![CDATA[tutorials]]></category>
		<category><![CDATA[database]]></category>
		<category><![CDATA[import]]></category>
		<category><![CDATA[large files]]></category>
		<category><![CDATA[slider]]></category>

		<guid isPermaLink="false">http://dev.eek.be/?p=548</guid>
		<description><![CDATA[I had to import a SQL-file with a size of over 20Gb into a mysql-database but ran into a lot of problems doing this. I don&#8217;t have the best server in the world so this import takes about 8 hours to complete and I noticed that after a few hours the import stopped or my ]]></description>
			<content:encoded><![CDATA[<p>I had to import a SQL-file with a size of over 20Gb into a mysql-database but ran into a lot of problems doing this.<br />
I don&#8217;t have the best server in the world so this import takes about 8 hours to complete and I noticed that after a few hours the import stopped or my connection broke down so I had to start the import again. I&#8217;ve tried it for 5 times without succes and got a little frustrated so I decided to write a little and simple PHP-script to split this large file into smaller pieces so I could import these pieces and eventually easily resume the script after the connection broke down.</p>
<p><b>Splitting the sql-file</b><br />
I just read every line in with PHP and when I encounter a comment (&#8211; ), I create a new file.</p>
<pre class="prettyprint">
//Set time limit to one hour
set_time_limit(3600);

$dir = '/path/to/store/splits';
$file = '/path/to/large/sql/file.sql';

$i = 1000000;
//open large sql-file
$handle = fopen($file,"r");
if($handle) {
	while(($buffer = fgets($handle)) !== false) {
		//read line and append it to the file with name $i.sql
		$newfile = fopen($dir . '/' .$i . '.sql', 'a+');
		fwrite($newfile,$buffer);
		//if a comment is found, create a new file
		if(substr($buffer,0,3) === '-- '){
			$i++;
		}
	}
fclose($handle);
}
</pre>
<p>Splitting this 20Gb file took about 10 minutes on my server and created approximately 500 files. The largest file was 5Gb and I know I could import a 5Gb-file without a problem.</p>
<p><b>Importing the sql-file</b><br />
Importing the file was pretty simple. I wrote a script that loops over all the files and imports them into the database.</p>
<pre class="prettyprint">
set_time_limit(3600000);

//The directory containing the splitted files
$dir = '/path/to/store/splits';

$files = scandir($dir);

ob_start();
//loop over the files and import them into the database
foreach($files as $file){
	echo $dir . $file . "\n";
	system("mysql -u mysql_username -pmysql_password mysql_databasename < $dir/$file");
	ob_flush();
}
</pre>
<p>In my case, it took more than 8 hours to completely import those database but with the help of these 2 scripts, it worked like a charm.</p>
]]></content:encoded>
			<wfw:commentRss>http://dev.eek.be/2011/04/split-and-import-a-very-large-database-with-php/feed/</wfw:commentRss>
		<slash:comments>6</slash:comments>
		</item>
		<item>
		<title>Testing the new mod_pagespeed for Apache 2 from Google</title>
		<link>http://dev.eek.be/2010/11/testing-the-new-mod_pagespeed-for-apache-2-from-google/</link>
		<comments>http://dev.eek.be/2010/11/testing-the-new-mod_pagespeed-for-apache-2-from-google/#comments</comments>
		<pubDate>Fri, 05 Nov 2010 21:33:34 +0000</pubDate>
		<dc:creator>Wim</dc:creator>
				<category><![CDATA[linux]]></category>
		<category><![CDATA[PHP]]></category>
		<category><![CDATA[servers]]></category>
		<category><![CDATA[testing]]></category>
		<category><![CDATA[ubuntu]]></category>
		<category><![CDATA[apache]]></category>
		<category><![CDATA[configuration]]></category>
		<category><![CDATA[Google]]></category>
		<category><![CDATA[modules]]></category>
		<category><![CDATA[mod_pagespeed]]></category>
		<category><![CDATA[mysql]]></category>
		<category><![CDATA[server]]></category>
		<category><![CDATA[speed]]></category>
		<category><![CDATA[tutorial]]></category>

		<guid isPermaLink="false">http://dev.eek.be/?p=427</guid>
		<description><![CDATA[Google released a new module for Apache2, mod_pagespeed, yesterday. They claim that a webpage can load 50% faster with the module installed. I&#8217;m a little bit sceptic about that claim so it&#8217;s time to give it a little test. For this test, I&#8217;ve set up a server (1 intel processor/512MB ram/12MB video memory) with Ubuntu ]]></description>
			<content:encoded><![CDATA[<p>Google released a new module for Apache2, <a href="http://googlecode.blogspot.com/2010/11/make-your-websites-run-faster.html" target="_blank">mod_pagespeed</a>, yesterday. They claim that a webpage can load 50% faster with the module installed.<br />
I&#8217;m a little bit sceptic about that claim so it&#8217;s time to give it a little test.</p>
<p>For this test, I&#8217;ve set up a server (1 intel processor/512MB ram/12MB video memory) with Ubuntu Server 10.10 (32 bit).<br />
I could test some static resources but nowadays, particularly all websites are dynamic ones so I&#8217;ve installed Mysql/PHP and of course Apache2, all with the default settings, on my server.<br />
Next, I&#8217;ve downloaded WordPress and installed it on the server, changed the appearance to &#8216;pixel theme&#8217; and added a few posts with images in.</p>
<p>Time to test <img src='http://dev.eek.be/wp-includes/images/smilies/icon_smile.gif' alt=':-)' class='wp-smiley' /> </p>
<p><span id="more-427"></span></p>
<p>I&#8217;m not going to do fancy stuff, the module is installed with the default options enabled and I&#8217;ll just use Firebug to do the tests.</p>
<p>First the tests without the module installed:<br />
<a href="http://dev.eek.be/wp-content/uploads/2010/11/mod_pagespeed2.jpg"><img class="aligncenter size-medium wp-image-436" title="mod_pagespeed " src="http://dev.eek.be/wp-content/uploads/2010/11/mod_pagespeed2-300x80.jpg" alt="" width="300" height="80" /></a></p>
<table id="wp-table-reloaded-id-2-no-1" class="wp-table-reloaded wp-table-reloaded-id-2">
<thead>
<tr class="row-1 odd">
<th class="column-1"></th>
<th class="column-2">pageRequests</th>
<th class="column-3">pagesize (KB)</th>
<th class="column-4">Loadtime (ms)</th>
</tr>
</thead>
<tfoot>
<tr class="row-12 even">
<th class="column-1">Average</th>
<th class="column-2">14</th>
<th class="column-3">354.9</th>
<th class="column-4">480</th>
</tr>
</tfoot>
<tbody class="row-hover">
<tr class="row-2 even">
<td class="column-1">Test 1</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">504</td>
</tr>
<tr class="row-3 odd">
<td class="column-1">Test 2</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">471</td>
</tr>
<tr class="row-4 even">
<td class="column-1">Test 3</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">596</td>
</tr>
<tr class="row-5 odd">
<td class="column-1">Test 4</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">459</td>
</tr>
<tr class="row-6 even">
<td class="column-1">Test 5</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">480</td>
</tr>
<tr class="row-7 odd">
<td class="column-1">Test 6</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">479</td>
</tr>
<tr class="row-8 even">
<td class="column-1">Test 7</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">446</td>
</tr>
<tr class="row-9 odd">
<td class="column-1">Test 8</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">457</td>
</tr>
<tr class="row-10 even">
<td class="column-1">Test 9</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">460</td>
</tr>
<tr class="row-11 odd">
<td class="column-1">Test 10</td>
<td class="column-2">14</td>
<td class="column-3">354.9</td>
<td class="column-4">444</td>
</tr>
</tbody>
</table>
<p>
The results with the module installed:<br />
<a href="http://dev.eek.be/wp-content/uploads/2010/11/mod_pagespeed12.jpg"><img src="http://dev.eek.be/wp-content/uploads/2010/11/mod_pagespeed12-300x69.jpg" alt="" title="mod_pagespeed" width="300" height="69" class="aligncenter size-medium wp-image-450" /></a></p>
<table id="wp-table-reloaded-id-3-no-1" class="wp-table-reloaded wp-table-reloaded-id-3">
<thead>
<tr class="row-1 odd">
<th class="column-1"></th>
<th class="column-2">pageRequests</th>
<th class="column-3">pagesize (KB)</th>
<th class="column-4">Loadtime (ms)</th>
</tr>
</thead>
<tfoot>
<tr class="row-12 even">
<th class="column-1">Average</th>
<th class="column-2">12</th>
<th class="column-3">349.7</th>
<th class="column-4">421.8</th>
</tr>
</tfoot>
<tbody class="row-hover">
<tr class="row-2 even">
<td class="column-1">Test 1</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">451</td>
</tr>
<tr class="row-3 odd">
<td class="column-1">Test 2</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">417</td>
</tr>
<tr class="row-4 even">
<td class="column-1">Test 3</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">406</td>
</tr>
<tr class="row-5 odd">
<td class="column-1">Test 4</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">437</td>
</tr>
<tr class="row-6 even">
<td class="column-1">Test 5</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">403</td>
</tr>
<tr class="row-7 odd">
<td class="column-1">Test 6</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">463</td>
</tr>
<tr class="row-8 even">
<td class="column-1">Test 7</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">393</td>
</tr>
<tr class="row-9 odd">
<td class="column-1">Test 8</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">436</td>
</tr>
<tr class="row-10 even">
<td class="column-1">Test 9</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">383</td>
</tr>
<tr class="row-11 odd">
<td class="column-1">Test 10</td>
<td class="column-2">12</td>
<td class="column-3">349.7</td>
<td class="column-4">429</td>
</tr>
</tbody>
</table>
<p><script type="text/javascript"><!--
google_ad_client = "pub-1041875485625119";
/* 468x60, gemaakt 5-11-10 */
google_ad_slot = "2519777813";
google_ad_width = 468;
google_ad_height = 60;
//-->
</script><br />
<script type="text/javascript"
src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
</script></p>
<p>Mod_pagespeed_statistics:</p>
<table id="wp-table-reloaded-id-4-no-1" class="wp-table-reloaded wp-table-reloaded-id-4">
<tbody class="row-hover">
<tr class="row-1 odd">
<td class="column-1">resource_fetches</td>
<td class="column-2">0</td>
</tr>
<tr class="row-2 even">
<td class="column-1">total_page_load_ms</td>
<td class="column-2">3240</td>
</tr>
<tr class="row-3 odd">
<td class="column-1">page_load_count</td>
<td class="column-2">10</td>
</tr>
<tr class="row-4 even">
<td class="column-1">cache_extensions</td>
<td class="column-2">30</td>
</tr>
<tr class="row-5 odd">
<td class="column-1">not_cacheable</td>
<td class="column-2">0</td>
</tr>
<tr class="row-6 even">
<td class="column-1">css_file_count_reduction</td>
<td class="column-2">0</td>
</tr>
<tr class="row-7 odd">
<td class="column-1">css_filter_files_minified</td>
<td class="column-2">10</td>
</tr>
<tr class="row-8 even">
<td class="column-1">css_filter_minified_bytes_saved</td>
<td class="column-2">40</td>
</tr>
<tr class="row-9 odd">
<td class="column-1">css_filter_parse_failures</td>
<td class="column-2">10</td>
</tr>
<tr class="row-10 even">
<td class="column-1">css_elements</td>
<td class="column-2">0</td>
</tr>
<tr class="row-11 odd">
<td class="column-1">image_inline</td>
<td class="column-2">10</td>
</tr>
<tr class="row-12 even">
<td class="column-1">image_rewrite_saved_bytes</td>
<td class="column-2">0</td>
</tr>
<tr class="row-13 odd">
<td class="column-1">image_rewrites</td>
<td class="column-2">20</td>
</tr>
<tr class="row-14 even">
<td class="column-1">javascript_blocks_minified</td>
<td class="column-2">0</td>
</tr>
<tr class="row-15 odd">
<td class="column-1">javascript_bytes_saved</td>
<td class="column-2">0</td>
</tr>
<tr class="row-16 even">
<td class="column-1">javascript_minification_failures</td>
<td class="column-2">0</td>
</tr>
<tr class="row-17 odd">
<td class="column-1">javascript_total_blocks</td>
<td class="column-2">0</td>
</tr>
<tr class="row-18 even">
<td class="column-1">url_trims</td>
<td class="column-2">0</td>
</tr>
<tr class="row-19 odd">
<td class="column-1">url_trim_saved_bytes</td>
<td class="column-2">0</td>
</tr>
<tr class="row-20 even">
<td class="column-1">resources_404_count</td>
<td class="column-2">0</td>
</tr>
<tr class="row-21 odd">
<td class="column-1">slurp_404_count</td>
<td class="column-2">0</td>
</tr>
<tr class="row-22 even">
<td class="column-1">serf_fetch_requests_count</td>
<td class="column-2">0</td>
</tr>
<tr class="row-23 odd">
<td class="column-1">serf_fetch_bytes_count</td>
<td class="column-2">0</td>
</tr>
<tr class="row-24 even">
<td class="column-1">serf_fetch_time_duration_ms</td>
<td class="column-2">0</td>
</tr>
<tr class="row-25 odd">
<td class="column-1">serf_fetch_cancel_count</td>
<td class="column-2">0</td>
</tr>
</tbody>
</table>
<p>
Conclusion:</p>
<ul>
<li>The amount of requests have been reduced in this case with 2 (<span style="color: #99cc00;">-15%</span>)</li>
<li>The pagesize has been reduced with 5.2KB (<span style="color: #99cc00;">-1.5%</span>)</li>
<li>The loadtime is shorter when mod_pagespeed is enabled (<span style="color: #99cc00;">-13%</span>)</li>
</ul>
<p>The 50% profit claim is a little bit optimistic because the bottleneck in the pageload is the dynamic content and that&#8217;s a part where this module doesn&#8217;t do anything. But in general, the module does a reasonably good job. I get a 13% profit with mostly default values, a good configured server should propably even do better.<br />
I will definitely test this further and maybe put it one day on a production server.</p>
<p>The config used in this test looks like this:</p>
<pre class="prettyprint">
&lt;IfModule pagespeed_module&gt;
    SetOutputFilter MOD_PAGESPEED_OUTPUT_FILTER
     ModPagespeed on
     ModPagespeedUrlPrefix "http://localhost/"
     ModPagespeedFileCachePath "/var/mod_pagespeed/cache/"
     ModPagespeedGeneratedFilePrefix "/var/mod_pagespeed/files/"
     ModPagespeedRewriteLevel CoreFilters
     ModPagespeedDomain localhost
     ModPagespeedEnableFilters add_instrumentation
&lt;IfModule&gt;
</pre>
<p>My full configuration file can be downloaded <a href='http://dev.eek.be/2010/11/testing-the-new-mod_pagespeed-for-apache-2-from-google/mod_pagespeed/' rel='attachment wp-att-456'>here</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://dev.eek.be/2010/11/testing-the-new-mod_pagespeed-for-apache-2-from-google/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>Backing up Mysql with Java and PHP</title>
		<link>http://dev.eek.be/2010/04/backing-up-mysql-with-java-and-php/</link>
		<comments>http://dev.eek.be/2010/04/backing-up-mysql-with-java-and-php/#comments</comments>
		<pubDate>Sun, 04 Apr 2010 09:35:26 +0000</pubDate>
		<dc:creator>Wim</dc:creator>
				<category><![CDATA[java]]></category>
		<category><![CDATA[PHP]]></category>
		<category><![CDATA[tutorials]]></category>
		<category><![CDATA[backup]]></category>
		<category><![CDATA[dump]]></category>
		<category><![CDATA[howto]]></category>
		<category><![CDATA[mysql]]></category>
		<category><![CDATA[mysqldump]]></category>
		<category><![CDATA[programming]]></category>

		<guid isPermaLink="false">http://dev.eek.be/?p=310</guid>
		<description><![CDATA[mysqldump is an effective tool to backup MySQL databases. In normal cases, it&#8217;s done trough a command line commando but it can be used programmatically. It&#8217;s a very easy script and it could be useful. (don&#8217;t have to log in into console to take a backup, create an auto backup on deployment, &#8230;) It will ]]></description>
			<content:encoded><![CDATA[<p>mysqldump is an effective tool to backup MySQL databases. In normal cases, it&#8217;s done trough a command line commando but it can be used programmatically.<br />
It&#8217;s a very easy script and it could be useful. (don&#8217;t have to log in into console to take a backup, create an auto backup on deployment, &#8230;)<br />
It will work on Windows and Linux as long as mysql is in your PATH.</p>
<pre class="prettyprint">
package be.eek.test.mysql;

import java.io.IOException;
import java.io.InputStream;

public class MysqlBackup {

    /**
    main method is just to create a working example
    **/
    public static void main(String[] args) throws IOException {
        new MysqlBackup().MakeBackup();
    }

    public void MakeBackup() throws IOException {
        String dump = "mysqldump "      //Path to mysql
                + "--host=localhost "       //Mysql hostname
                + "--port=3306 "            //Mysql portnumber
                + "--user=root "            //Mysql username
                + "--password=test "        //Mysql password
                + "--add-drop-table "       //Add a DROP TABLE statement before each CREATE TABLE statement
                + "--add-drop-database "    //Add a DROP DATABASE statement before each CREATE DATABASE statement
                + "--complete-insert "      //Use complete INSERT statements that include column names.
                + "--extended-insert "      //Use multiple-row INSERT syntax that include several VALUES lists
                + "test";                   //Mysql databasename

        Process run = Runtime.getRuntime().exec(dump);

        InputStream in = run.getInputStream();

        int nextChar;
        StringBuffer sb = new StringBuffer();

        while ((nextChar = in.read()) != -1) {
            sb.append((char) nextChar);
        }

        //Here, you can for example write it to a file and save it
        System.out.println(sb);
    }
}
</pre>
<p>The same thing is also possible in php:</p>
<pre class="prettyprint lang-php">
$dump = "mysqldump "
        . "--host=localhost "       //Mysql hostname
        . "--port=3306 "            //Mysql portnumber
        . "--user=root "            //Mysql username
        . "--password=test "        //Mysql password
        . "--add-drop-table "       //Add a DROP TABLE statement before each CREATE TABLE statement
        . "--add-drop-database "    //Add a DROP DATABASE statement before each CREATE DATABASE statement
        . "--complete-insert "      //Use complete INSERT statements that include column names.
        . "--extended-insert "      //Use multiple-row INSERT syntax that include several VALUES lists
        . "test";                   //databasename

$backup = system($dump);
echo $backup;
die();
</pre>
]]></content:encoded>
			<wfw:commentRss>http://dev.eek.be/2010/04/backing-up-mysql-with-java-and-php/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

