Today I had to rethink how I serve data to my users.

According to an answer posted on StackOverflow,  gzip or deflate compression should be avoided when serving text-based files.

While compressing data to decrease bandwidth and download times generally seems like a great idea, there are a number of problems which have surfaced during the evolution of on-the-fly compression over the wire. After reading the article Lose the wait, I thought more of why I had implemented used mod_gzip in the past and started to realize that the benefits aren’t particularly worth the effort or expense.

• Internet Explorer has a history of issues with receiving compressed payloads — so much history that all gzip/deflate HOWTOs have a line disabling compression for IE, based upon user agent evaluation.
• Uses extra CPU cycles to compress data on each request.
• Requires another module to be loaded into memory (mod_gzip, mod_deflate, etc)
• SSL (HTTPS) connections cannot be compressed.

A better approach is to combine technologies like browser caching and local storage mechanisms, server-side expires tags, and code “minification” to mitigate the need for on-the-fly compression by addressing the underlying issue: management of client-to-server requests.

All that said, compressed http payloads still have a place on the web. For example, Reddit, Digg, Slashdot, or any high-traffic, predominantly text-based site with a large volume of traffic from modern, non-IE web browsers.

What’s so great about Git?

I had planned to sit down this evening and write an article extolling the virtues of git and version control, but I’ve discovered loads of articles who have done it much better than I ever could. So instead of reinventing the wheel, I will merely redirect you to to these articles for an introduction to git, then make a few additional points.

Required reading:
Designers read Git: A Designer’s Perspective.
Developers read the article above and Git for the nervous developer.

Extracurricular:
A SlideShare presentation on the more technical aspects of Git.

Git options on Mac OS X: 

• GitTower – Commercial gui client ($69)
• git-osx-installer – open-source installation of command-line git
• port install git-core – using mac ports (formerly darwin) package manager

Git options for Windows users:

• TortiseGit - open-source client with Windows Explorer integration
• MSysGit - open-source command-line client
• SmartGit - commercial ($69)

Git repository hosting:

• Github – Packages geared toward many users and better collaboration, less repositories
• Beanstalk – Repository hosting with packages geared toward fewer developers, but more repositories per account tier.
• Unfuddle – Git and SVN hosting, geared toward project managements (has wikis, gantt charts, etc). Silly name.

More resources:

Pro Git – A free git bible.

My experience:

I first noticed that my mouse pointer will jump around the screen while moving it. That is, I would move the mouse a short distance but it does not move (seems “frozen”), then appears on the screen having covered a greater distance than it should have (100px move, appears 250px away [arbitrary values]), with no movement animation.

This seems to happen when various components in my system are under some sort load. It reminds me of older PCs and how human interface devices would take a back-seat when the system bus became overloaded.

I have noticed the problem occur during the following system activities:

1. Dropbox displaying a growl notification about files being removed from my system (Growl, video, disk?)
2. Saving out files in photoshop (video?, disk)

What I know it is not:

1. Solely Disk I/O related. I have run a command like “dd if=/dev/random of=/tmp/largefile bs=1024 count=4000000″ and moved the mouse all around the screen.
2. Solely video-related. I can play games just fine, though the pointer does become noticeably more jumpy.

I expect the problem is related to multiple components.

My current system configuration:

• 21″ iMac (Mid-2010)
• 8GB ram
• Additional 20″ monitor (mini-to-dvi adapter)
• Logitech MX518 mouse
• Apple USB keyboard

I decided to finish setting up VPN access to the office and part of the process was to change our local subnet of “192.168.1.0″ to “192.168.x.0″ (for x > 1) to avoid ip address conflicts with most home routers. It’s not a difficult process, but I did encounter a few stumbling blocks along the way and I have noted them below.

Obvious Tip of the Day: Change your TCP/IP settings on all devices that have IP settings manually configured.

Our OSX Lion server has its IP address manually assigned on the server itself, and luckily I was smart enough to have changed the IP address on the machine before making the subnet change. Unfortunately, I did not remember about the virtual machines that live in Virtualbox on the Lion server. After reconfiguring the router with the new subnet information I was left without my dev linux servers or the ability to access the internet. Here’s why:

Virtualbox does not like moving subnets

After rebooting the Lion server, my VMs (which normally start at boot) did not come back online and I was unable to connect to them using the Virtualbox GUI, even though the GUI suggested they were stopped. I believe I was getting an “access denied” error.

I unloaded the VMs .plist from OSX’s LaunchDaemon and started the VM manually via the VBox GUI, but once the machine had loaded to the login prompt I couldn’t type into the console! Also, when checking the settings for each VM showed a “Invalid Settings Detected” message at the bottom of the dialog.

Eventually I found that the problem was related to the subnet change having hosed VirtualBox’s virtual network interfaces.

The Fix

I added a new virtual interface by going to Virtualbox Preferences, Network, and clicking the plus (+) button to add a new interface for each VM. Not only did this give me the ability to interact with each VM via the VBox GUI, but it also allowed the VMs to reconnect to the network (obviously). Woohoo!

Except the ugly “Invalid settings detected” error was still displayed in the settings dialog for my VMs, which also has the nasty side-effect of disabling the OK button so you cannot change any settings.

The Fix, part 2

More googling research eventually let me to a forum post that solved my problem.

Run the following command:

$ VBoxManage showvminfo “Your VM Name”

If you see the following line:

VRDE:            enabled (Address 0.0.0.0, Ports , MultiConn: off, ReuseSingleConn: off, Authentication type: null)

Run this command:

$ VBoxManage modifyvm “your VM Name” –vrde off

Voila! Error message gone and you can edit your settings again.

Of course this is a band-aid fix and you should really understand what VRDE does before leaving it off indefinitely, but it’s Friday afternoon and I’m going home. See ya!

I installed OSX Lion onto my work machine this week and, while I do love the new features, I’ve been having some problems. Most notably, once I switched away from the carpel-tunnel-causing Magic Mouse to the luscious Logitech MX518 I have been experiencing the same “pointer skip” that many other people are complaining about.

After some research it seems like the problem is related to mouse acceleration during high CPU usage, and it seems that the only way to mitigate the issue is to turn off mouse acceleration. I found a piece of software that allowed me do this, and it seems to have helped stop the pointer from jumping around as much but the problem isn’t completely solved. Obviously turning off isn’t a solution.

Here are two solutions to turning off mouse acceleration:

1. Command line app [source]

$ curl -O http://ktwit.net/code/killmouseaccel
$ chmod +x killmouseaccel
$ ./killmouseaccel mouse

2. Use the app Kill Mouse Accel.app.

Try these fixes at your own risk. I used the option #1 above and it worked just fine. I hope this gets addressed in the near future.. it’s really odd mousing around the screen without acceleration.

Given that I built this system 2 weeks ago it isn’t quite “new” but it mines at around 750-800 Mhash/sec, or 1 bitcoin every 2 days (via deepbit). I think that’s pretty respectable.

Here’s the rig that I’ve christened “Newbie”:

• 2x ATI Radeon HD 5850 video cards (w/ CrossfireX)
• Intel i3-2100 (Dual core, 3.1 GHz)
• Asus P8P67LE motherboard (w/ 2 PCIE-16 slots)
• 4GB DDR3 RAM
• 640GB WD HDD
• Antec case

My first calculations were that the machine would paid for itself in approximately 3 months, but due to several factors affecting the BTC exchange (increased difficulty, less interest, falling USD, etc) I’m not sure where this will take the market. Either way, it’s still an interesting experiment!

I’m going to post some mining statistics about the rig’s once I get a chance.

...
		// Miscellaneous parameters
		$depth = $this->EE->TMPL->fetch_param('limit_depth', 1);
		$exclude = $this->EE->TMPL->fetch_param('exclude', NULL);
		$mode = $this->EE->TMPL->fetch_param('mode', 'sub');
		$prepend = $this->EE->TMPL->fetch_param('prepend', NULL);
...

And before the end of the function, replace the line setting $html with the following highlighted line:

...
		if ($entry_id === FALSE)
			$entry_id = 0;

		$html = $this->sql->generate_nav($entry_id, $current_id, $depth, $exclude, $mode, $prepend);

		return $html;
	}
...

Now open sql.structure.php.

Locate the generate_nav() function (around line 479) and change the function definition to the first highlighted line below. Also update the function call to get_selective_data() (around line 489) to pass through that $prepend value as shown in the second line below.

...
	/**
	 * Get the HTML code for an unordered list of the tree
	 * @return string HTML code for an unordered list of the whole tree
	 */
	function generate_nav($entry_id, $current_id = FALSE, $depth = 1, $exclude = NULL, $mode, $prepend=null)
	{

		$html = '';
		$separator = $this->EE->config->item('word_separator') != "dash" ? '_' : '-';

		// Fallback to entry_id if no current_id (e.g. sitemap usage)
		if ($current_id === FALSE)
			$current_id = $entry_id;

		$pages = $this->get_selective_data($entry_id, $depth, $exclude, $current_id, $mode, $prepend);
		$pages = $this->add_attributes($pages, $current_id, $mode);
...

Now we update get_selective_data() (around line 139) to make use of the passed data:

	/**
	 * Get selective data on all Structure Channels
	 * The Godfather of all Structure queries
	 *
	 * "I'm gonna make you a query you can't refuse."
	 *
	 * @return array
	 */
	function get_selective_data($parent_id = NULL, $depth = 1, $exclude = NULL, $current_id = NULL, $mode = NULL, $prepend = NULL)
	{
		$pages = $this->get_site_pages();
...

Please take a moment and go back to thoroughly review the inline comment.

Finally, in the same get_selective_data() function find the code which sets “$data[$row['entry_id']]['uri']” (around line 286) and replace it with the highlighted code:

...
			// build out the main data array
			foreach ($page_data as $row)
			{
				$data[$row['entry_id']] = $row;
				//$data[$row['entry_id']]['uri'] = $this->EE->functions->create_page_url($pages['url'] .'/'. $prepend, $pages['uris'][$row['entry_id']]);
				$data[$row['entry_id']]['uri'] = $this->EE->functions->create_page_url($pages['url'] .'/'. $prepend, $pages['uris'][$row['entry_id']]);
				$data[$row['entry_id']]['slug'] = $pages['uris'][$row['entry_id']];
				$data[$row['entry_id']]['classes'] = array();
				$data[$row['entry_id']]['ids'] = array();
...

That does it. You should now be able to add an arbitrary string to the beginning of the path portion of a URL.

Example code:

...
{exp:structure:nav mode="sub" lang="{site_lang}" start_from="/{segment_1}" }
...

Note: {site_lang} is a snippet I’ve created in the EE back end that always echos the current site “stub”. Also, the “start_from” attribute isn’t required.

You can use the same principles in this post to prepend strings to the Structure breadcrumbs and other parts of Structure. Please be aware, however, if you modify the core code the Structure guys are not going to be able to help you with any problems you might encounter.

Sure, MobileMe automagically configured the baseline (settings, accounts, etc) for me but I’m still left with a number of particulars.. namely configuring my virtual workspace: editors, browsers, scripts, etc. Below is a list of all the custom add-ons I’ve installed to make my copy of TextMate wicked awesome:

Essentials:

• MissingDrawer (gh) – This plugin provides Xcode-like project window interface without drawer and adds “Open Terminal Here” button to the file list’s button panel.

Awesome TextMate bundles:

• Javascript Tools bundle (gh) - Essential Tools for developing Javascript in TextMate
• mHTML bundle (gh) - TextMate XHTML Bundle that extends the functionality of the default HTML bundle. (screencast)
• mCSS bundle (gh) - TextMate CSS Bundle that extends the default bundle with a million snippets. (screencast)
• jQuery bundle (gh) – jQuery bundle to aid in jQuery development.
• TextMade MiniMap (gh) - Display a “mini-map” or preview for a file in a pane.
• nice_find (gh) – Written in Objective C and interacts directly with TextMate. Uses grep (or git-grep where available). As a result it is faster, more intuitive, and has a native look and feel that is the envy of all the other grep bundles.
• TmCodeBrowser -
• ReMate - ReMate is a TextMate plugin which can be used to disable the automatic project tree refresh when the application regains focus.
• git-tmbundle (gh) – Built-in git support for TextMate

Themes:

• IR_Black.tmTheme – Black theme that has different colors for a wide variety of items, with coordinated colors between like items. (Ruby, Java, HTML, XML)
• Clouds – Light coloured, minimal theme. Very nice. (HTML, CSS, Javascript, PHP)

Other Resources:

• The Hidden Magic in TextMate
• http://wiki.macromates.com/Main/Plugins

Out-of-the-box page hierarchy on the Structure control panel

Now for all of our Expression Engine sites we immediately install Structure, a badass module that completely reinvents how EE handles pages. Basically, it’s the best thing ever.. it’s got an intuitive design and saves us (me) from messing around trying to hack a “page structure” into EE, and lists site content pages in a hierarchy on a single view on the admin control panel which makes it super-easy to find or edit pages. Awesome! But even with all those perks, managing all that content in Structure became a bit cumbersome as everything in our copy was referenced by the “internal address” I mentioned above, not by the page titles that appear on the Structure control panel.

The solution? Add our internal document address system custom field to the Structure control panel page!

After adding custom field data to the page hierarchy

Keep in mind that this should work for any custom field attached to a Structure page (provided the field exists), but I’m not making any guarantees or warranty that this will work.. even though it should. Basically, it works on my machine using an install of ExpressionEngine 2.1.3 but, as with everything in life, YMMV.

Further Warning: This will modify the core code for Structure. Don’t expect Travis or Jack (the guys behind Structure) to support your extension if these modifications break your site, cause you mental anguish, or murder your goldfish. You’ve been warned.

Step 0:

FOR GOODNESS SAKE, BACK UP EVERYTHING.

Step 1:

Get the Field ID for the custom field you want to include on the Structure Control Panel page. You can get this by going to Admin -> Channel Administration -> Custom Fields. Click “Add/Edit Custom Fields” for the appropriate Field Group, then click on the custom field you will be using.

Now look at the URL, you should see an address like this:
http://csrdev.powershiftermedia.com/admin/index.php?S=1234567890deadbeef&D=cp&C=admin_content&M=field_edit&field_id=21&group_id=5

Write down the number that comes after “field_id=” in the URL; in this case it’s “21″. We will be using this later.

Step 2

Open the file “expressionengine/third_party/stricture/sql.structure.php” and find the “get_data()” function, on-or-around line 83. We are going to update the SQL query used by this function.

The SQL query at the time of writing is the following:

$sql = "SELECT node.*, (COUNT(parent.entry_id) - 1) AS depth, expt.title, expt.status
FROM exp_structure AS node
INNER JOIN exp_structure AS parent
ON node.lft BETWEEN parent.lft AND parent.rgt
INNER JOIN exp_channel_titles AS expt
ON node.entry_id = expt.entry_id
WHERE parent.lft > 1
AND node.site_id = {$this->site_id}
AND parent.site_id = {$this->site_id}
GROUP BY node.entry_id
ORDER BY node.lft";

And we’re going to update it to the following:

$sql = "SELECT node.*, (COUNT(parent.entry_id) - 1) AS depth, expt.title, expt.status
,expd.field_id_21 AS content_address
FROM exp_structure AS node
INNER JOIN exp_structure AS parent
ON node.lft BETWEEN parent.lft AND parent.rgt
INNER JOIN exp_channel_titles AS expt
ON node.entry_id = expt.entry_id
INNER JOIN exp_channel_data as expd ON node.entry_id = expd.entry_id
WHERE parent.lft > 1
AND node.site_id = {$this->site_id}
AND parent.site_id = {$this->site_id}
GROUP BY node.entry_id
ORDER BY node.lft";

Notice the additions I’ve made on the 2nd and 8th lines. Let me explain:

• Line 2: Replace “21″ with the Field ID you noted from Step 1. This will return the data you care about. You can also add “AS your_field_name” bit to your SQL to help you identify the field in the next step.
• Line 8: This INNER JOIN connects the page title we’re returning to the custom field data for the page. You don’t need to edit this line.

Save and close the file.

Step 3:

Open the file “expressionengine/third_party/structure/views/index.php” and find following code (on or around line 69):

<a class="page-edit" href="<?=$edit_url;?>"><?=$page['title'];?></a>

Now update the code to add in your custom field data, similar to the following:

<a class="page-edit" href="<?=$edit_url;?>"><?=$page['content_address'];?>&nbsp;<?=$page['title'];?></a>

Save and close the file. Done!


2011/04/06 14:17:36 [error] 32174#0: *9366 FastCGI sent in stderr: "PHP Warning: Unknown: Unable to allocate memory for pool. in Unknown on line 0
PHP Warning: require_once(): Unable to allocate memory for pool. in /PATH/TO/MYSITE/phpmyadmin/index.php on line 35
PHP Warning: require_once(): Unable to allocate memory for pool. in /PATH/TO/MYSITE/phpmyadmin/libraries/common.inc.php on line 53
PHP Warning: require_once(): Unable to allocate memory for pool. in /PATH/TO/MYSITE/phpmyadmin/libraries/Error_Handler.class.php on line 13
PHP Warning: require_once(): Unable to allocate memory for pool. in /PATH/TO/MYSITE/phpmyadmin/libraries/Error.class.php on line 13" while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www.example.com, request: "GET /phpmyadmin/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.example.com"


I corrected this by editing /etc/php.d/apc.ini and setting apc.shm_size=64M.

Next, I encountered the following error:

011/04/06 16:54:35 [error] 7399#0: *492 FastCGI sent in stderr: "PHP Fatal error: Class 'PMA_Message' not found in /PATH/TO/MYSITE/phpmyadmin/libraries/Message.class.php on line 732" while reading upstream, client: xxx.xxx.xxx.xxx, server: www.example.org, request: "GET /phpmyadmin/phpmyadmin.css.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.example.org"

I found a hint to fix the problem, and discovered that PHP packages installed by the IUS Community Project store sessions into /var/lib/php/session, which is owned by apache:apache. Since I don't use apache, I had changed both the PHP-FPM pool user and group to "nginx", and that change removed PHP-FPM's ability to write to sessions.

Lessons learned:

• PHP packages installed via the IUS Community Project attempt to save sessions to /var/lib/php/session, which is owned by apache:apache. Keep this in mind when modifying permissions or the accounts under which the services run.