Raising URI::InvalidURIError from a perfectly valid URI

I was puzzled by URI::parse raising an URI::InvalidURIError on a perfectly well formed URI recently.

[ruby]
URI::InvalidURIError: bad URI(is not URI?): http://practicalguile.com/articles?query=latest
from /opt/local/lib/ruby/1.8/uri/common.rb:436:in `split’
from /opt/local/lib/ruby/1.8/uri/common.rb:485:in `parse’
from (irb):2
from :0
[/ruby]

What’s not apparent in this exception message is that the url contained a trailing space and this was causing URI.parse to fail. The following specifications demonstrate how it can trigger this particular exception.

uri.spec.rb
[ruby]
require ‘rubygems’
require ‘spec’
require ‘uri’

describe URI do
it “should raise an InvalidURIException with leading whitespace in url” do
lambda{ URI.parse(‘ http://www.ruby-lang.org’) }.should raise_error(URI::InvalidURIError)
end

it “should raise an InvalidURIException with trailing whitespace in url” do
lambda{ URI.parse(‘http://www.ruby-lang.org ‘) }.should raise_error(URI::InvalidURIError)
end
end
[/ruby]

Running the spec will get you the result below.

ruby uri.spec.rb

..Finished in 0.030051 seconds

2 examples, 0 failures

Looking at the stacktrace in the exception, it’s being raised by URI.split after URI.parse is invoked with the offending URL.

RUBY_INSTALL/1.8/uri/common.rb

[ruby]
def self.parse(uri)
scheme, userinfo, host, port,
registry, path, opaque, query, fragment = self.split(uri)

if scheme && @@schemes.include?(scheme.upcase)
@@schemes[scheme.upcase].new(scheme, userinfo, host, port,
registry, path, opaque, query,
fragment)
else
Generic.new(scheme, userinfo, host, port,
registry, path, opaque, query,
fragment)
end
end
[/ruby]

Nothing weird happening in URI.parse, its a straightforward call to URI.split. So I’ll go into URI.split, comments removed for brevity.

[ruby]
def self.split(uri)
case uri
when ”
when ABS_URI
scheme, opaque, userinfo, host, port,
registry, path, query, fragment = $~[1..-1]

if !scheme
raise InvalidURIError,
“bad URI(absolute but no scheme): #{uri}”
end
if !opaque && (!path && (!host && !registry))
raise InvalidURIError,
“bad URI(absolute but no path): #{uri}”
end
when REL_URI
scheme = nil
opaque = nil

userinfo, host, port, registry,
rel_segment, abs_path, query, fragment = $~[1..-1]
if rel_segment && abs_path
path = rel_segment + abs_path
elsif rel_segment
path = rel_segment
elsif abs_path
path = abs_path
end
else
raise InvalidURIError, “bad URI(is not URI?): #{uri}”
end

path = ” if !path && !opaque # (see RFC2396 Section 5.2)
ret = [
scheme,
userinfo, host, port, # X
registry, # X
path, # Y
opaque, # Y
query,
fragment
]
return ret
end
[/ruby]

URI.split is matching the incoming url with an empty string as well as regular expressions for absolute and relative URIs. It’s obvious from the specifications earlier that urls with leading/trailing whitespace do not match any of these and the case statement raises InvalidURIError, with the rather misleading message.

The regexes used for matching absolute and relative URIs is shown below, if you really want to know.
[ruby]
require ‘uri’
include URI::REGEXP

ABS_URI
/^
([a-zA-Z][-+.a-zA-Zd]*): (?# 1: scheme)
(?:
((?:[-_.!~*'()a-zA-Zd;?:@&=+$,]|%[a-fA-Fd]{2})(?:[-_.!~*'()a-zA-Zd;/?:@&=+$,[]]|%[a-fA-Fd]{2})*) (?# 2: opaque)
|
(?:(?:
//(?:
(?:(?:((?:[-_.!~*'()a-zA-Zd;:&=+$,]|%[a-fA-Fd]{2})*)@)? (?# 3: userinfo)
(?:((?:(?:(?:[a-zA-Zd](?:[-a-zA-Zd]*[a-zA-Zd])?).)*(?:[a-zA-Z](?:[-a-zA-Zd]*[a-zA-Zd])?).?|d{1,3}.d{1,3}.d{1,3}.d{1,3}|[(?:(?:[a-fA-Fd]{1,4}:)*(?:[a-fA-Fd]{1,4}|d{1,3}.d{1,3}.d{1,3}.d{1,3})|(?:(?:[a-fA-Fd]{1,4}:)*[a-fA-Fd]{1,4})?::(?:(?:[a-fA-Fd]{1,4}:)*(?:[a-fA-Fd]{1,4}|d{1,3}.d{1,3}.d{1,3}.d{1,3}))?)]))(?::(d*))?))?(?# 4: host, 5: port) |
((?:[-_.!~*'()a-zA-Zd$,;+@&=+]|%[a-fA-Fd]{2})+) (?# 6: registry)
)
|
(?!//)) (?# XXX: ‘//’ is the mark for hostport)
(/(?:[-_.!~*'()a-zA-Zd:@&=+$,]|%[a-fA-Fd]{2})*(?:;(?:[-_.!~*'()a-zA-Zd:@&=+$,]|%[a-fA-Fd]{2})*)*(?:/(?:[-_.!~*'()a-zA-Zd:@&=+$,]|%[a-fA-Fd]{2})*(?:;(?:[-_.!~*'()a-zA-Zd:@&=+$,]|%[a-fA-Fd]{2})*)*)*)? (?# 7: path)
)(?:?((?:[-_.!~*'()a-zA-Zd;/?:@&=+$,[]]|%[a-fA-Fd]{2})*))? (?# 8: query)
)
(?:#((?:[-_.!~*'()a-zA-Zd;/?:@&=+$,[]]|%[a-fA-Fd]{2})*))? (?# 9: fragment)
$/xn

REL_URI
/^
(?:
(?:
//
(?:
(?:((?:[-_.!~*'()a-zA-Zd;:&=+$,]|%[a-fA-Fd]{2})*)@)? (?# 1: userinfo)
((?:(?:(?:[a-zA-Zd](?:[-a-zA-Zd]*[a-zA-Zd])?).)*(?:[a-zA-Z](?:[-a-zA-Zd]*[a-zA-Zd])?).?|d{1,3}.d{1,3}.d{1,3}.d{1,3}|[(?:(?:[a-fA-Fd]{1,4}:)*(?:[a-fA-Fd]{1,4}|d{1,3}.d{1,3}.d{1,3}.d{1,3})|(?:(?:[a-fA-Fd]{1,4}:)*[a-fA-Fd]{1,4})?::(?:(?:[a-fA-Fd]{1,4}:)*(?:[a-fA-Fd]{1,4}|d{1,3}.d{1,3}.d{1,3}.d{1,3}))?)]))?(?::(d*))? (?# 2: host, 3: port)
|
((?:[-_.!~*'()a-zA-Zd$,;+@&=+]|%[a-fA-Fd]{2})+) (?# 4: registry)
)
)
|
((?:[-_.!~*'()a-zA-Zd;@&=+$,]|%[a-fA-Fd]{2})+) (?# 5: rel_segment)
)?
(/(?:[-_.!~*'()a-zA-Zd:@&=+$,]|%[a-fA-Fd]{2})*(?:;(?:[-_.!~*'()a-zA-Zd:@&=+$,]|%[a-fA-Fd]{2})*)*(?:/(?:[-_.!~*'()a-zA-Zd:@&=+$,]|%[a-fA-Fd]{2})*(?:;(?:[-_.!~*'()a-zA-Zd:@&=+$,]|%[a-fA-Fd]{2})*)*)*)? (?# 6: abs_path)
(?:?((?:[-_.!~*'()a-zA-Zd;/?:@&=+$,[]]|%[a-fA-Fd]{2})*))? (?# 7: query)
(?:#((?:[-_.!~*'()a-zA-Zd;/?:@&=+$,[]]|%[a-fA-Fd]{2})*))? (?# 8: fragment)
$/xn
[/ruby]

Looks rather intimidating, doesn’t it? However, we’re more interested in the beginning and end of the regular expressions so its safe to ignore all the stuff in between. Narrowing our focus down to the regex anchors (^ and $), we can see that there is no matching of whitespace, thus preventing a valid URI from being matched in URI.split.

This all means that URI.split has a undocumented pre-condition on the uri parameter being stripped of any whitespace around it.

Using Factories for Rails Fixtures and Test Doubles

Chris Wanstrath has written about making Rails fixtures less painful than they need to be with the FixtureScenarios plugin. Personally, I prefer the Factory approach, nicely explained by Daniel Manges.

I’ve been using factory methods to create in-database ActiveRecord objects for a project that I’ve been working on in Bezurk. Reading Daniel’s article gave me a few ideas on improving the way I create fixtures and mocks. Since I’ve been using RSpec extensively in this project, I’ll present the examples in RSpec.

As the models evolve with the design and its behaviour change accordingly, there is a need to go through all the specifications that create this model and make sure that its created in a valid state. This is more pronounced with the use of test doubles, the test doubles also need to have its method stubs changed to reflect the latest state of the model that its is representing. I happen to make much use of test doubles for test isolation, so trying to manage all these objects became an exercise in patience. As it was getting painful, It’s time to change the way I create these models and test doubles.

As always, a layer of indirection will always go some way to solving a software problem. We introduce a Factory that encapsulates the creation of ActiveRecord objects by providing creation methods.

[ruby]
module FixtureFactory
def create_user(attributes = {})
User.create!(ModelAttributes.user(attributes))
end
end
[/ruby]
We’ll have a Factory for test doubles too.

[ruby]
module MockFactory
def mock_user(method_stubs = {})
mock_model(User, ModelAttributes.user(method_stubs))
end
end
[/ruby]
And the attributes for this model will be declared in a module that’s used by both Factories

[ruby]
module ModelAttributes
def self.user(attributes)
attributes.reverse_merge({:name => ‘doug’})
end
end
[/ruby]

The Factory modules are then included in Spec::Runnner

[ruby]
Spec::Runner.configure do |config|
include FixtureFactory
include MockFactory
end
[/ruby]

The objects can now be created using the factory methods available to all specifications.

[ruby]
doug = create_user
doppelganger = mock_user
[/ruby]

Update
Added links to Chris Wanstrath and Daniel Manges’ articles on managing Rails fixtures.

RSpec your functional tests

The release of RSpec 0.9.4 can be considered a landmark release of sorts. With the inclusion of Spec::Ui 0.2.0, RSpec can now perform functional testing of web applications within a browser. Out of the box support is limited to Watir/Internet Explorer for now as the Selenium RC custom matchers are not implemented yet. includes WATIR/Windows and Watirsafari/OSX. The custom matchers for Selenium RC are not implemented yet but its certainly possible to use Selenium with Spec::Ui, it just won’t read as nicely compared to WATIR. Spec::Ui also comes with a custom result formatter that packages a screenshot of the browser on spec failure, along with its HTML source in the report.

To use Watirsafari in your functional specs, you’ll need to install the gem for it: gem install watirsafari

spec_helper.rb should be updated to include the snippet below (code lifted from the original release announcement):

[ruby]
Spec::Runner.configure do |config|
config.include Spec::Matchers::Watir
end
[/ruby]

The specification will need before and after blocks like this:
[ruby]
before(:all) do
@browser = Watir::Browser.new
end

after(:each) do
# This is needed to make screenshots work
Spec::Ui::ScreenshotFormatter.browser = @browser
end

after(:all) do
@browser.kill! rescue nil
end
[/ruby]

There are a lot more useful information available in the examples supplied with Spec::Ui. Take a look at the samples to get up and running.

Update
My thanks to Aslak Hellesoy for correcting me on support for WATIR and Selenium.

Using Flash for background animation

I was working on a website recently that needed to display Flash animation beneath normal HTML content as part of its visual treatment. In the course of developing this, I found out that embedded Flash objects inserted into a HTML document will be displayed above all other content.

The easiest way to get around this is to enclose the Flash object in a another element, for example a <div> . This enclosing element will then have a value for z-index that is lower than the content that should be displayed above the Flash object.

[html]

Flash object markup

Actual Page Content

[/html]

The next step is to position the content and Flash object, there are 2 ways to “layer” the 2 elements on top of each other. The first method is to use absolute positioning, the second is to take advantage of negative margins. In my case, I went with the latter.I will pull div#content up towards the Flash object by declaring a negative top margin for #content.

[html]

Flash object markup

Actual Page Content

[/html]

In the course of testing, the page is rendered as intended in Internet Explorer 6/7 and Firefox. However, the Flash object will be displayed above content in Safari when any text is selected or the page is scrolled by the user. To get around this issue, enable transparency for the Flash object. Just take care to position content away from the non-transparent parts of the Flash animation or it will be hidden from view in Safari.

Using Textmate for Actionscript

I’ve never liked the default code editor that comes with the Flash IDE, so when the I discovered that I would need to develop actionscript code for a project, my first thought was to use Textmate instead. Incidentally, some people are already doing this.

I’ve used MTASC to compile Actionscript files before, the most important difference that surfaces when comparing it to Macromedia’s Actionscript compiler (MMC), which is the one that’s packaged in every installation of Flash, is that MTASC is much stricter on enforcing proper syntax. This is intended to reduce occurences of hard to find bugs that arise from improper scripting. The other feature that MTASC touts is its faster compile speed, however I’ve not experienced a visible variance between it and MMC. I would think that this speed improvement would be more apparent when compiling large projects consisting of more than 50 Actionscript files.

I’m using dirtystylus’ Textmate command for checking the syntax of my files, it took me a while to get it working as I organised the dependencies differently. Following my conventions in using MacPorts which is installed in /opt/local/, MTASC and XTrace were placed in /opt/local/managed/. The executable for MTASC was stored in /opt/local/managed/bin/ while XTrace.app was copied to the Applications folder. All supporting scripts were moved to /opt/local/managed/lib/.

This is the customised command that I’m using after trying for about 2 hours.

FLASHPLAYER=SAFlashPlayer
MTASC=/opt/local/managed/bin/mtasc
CLASSPATHS="-cp /opt/local/managed/lib/mtasc/std/ -cp /opt/local/managed/lib/mtasc/std8/ -cp /Users/douglas/Library/Application Support/Macromedia/Flash 8/en/Configuration/Classes/"
SOURCE="$TM_FILEPATH"
OUTPUT=test.swf
VERSION=8
TRACE="-trace com.mab.util.debug.trace"
TMPFILE=/tmp/as-compile.err
compileResult=`$MTASC -main "$SOURCE" -wimp -version $VERSION -strict $CLASSPATHS $TRACE 2&gt;&amp;1`
echo "

 body {margin: 10px; font-family: Monaco; font-size: 14px;} a {color: #000000; text-decoration: underline;} a:hover {color: #666666;}   function closeWin() { self.close(); } "
if test -n "$compileResult"
then
errorLine=`echo $compileResult | sed 's/.*:([0-9]*):.*/1/'`
echo "<a href="//$TM_FILEPATH&amp;line=$errorLine">$compileResult</a>";
else
echo "
fi

Singapore Web Standards Group

I enjoyed the first Singapore Web Standards Group meetup held at Raffles Girls School. Saw a few familiar faces and several new ones as well. It’s good to know that web standards are alive and kicking in Singapore. I really appreciate the effort given by Lucian who organised the meetup, and for the entertaining and informative presentations given by Coleman and Nick.

Although I thought the meetup could have done with more publicity, I only got to know of it from Sip Khoon a week before. I suppose it’s something that we can do to make people aware of the options available

Update
Ivan Lian has pictures of the meetup on Flickr. Check it out!

Fixing MacPort’s MySQL5 install

I had to reinstall OSX the other day as certain applications were crashing with alarming frequency. As I’ve installed DarwinPorts(now called MacPorts) for managing my software development libraries, I reinstalled the usual application via port install app. Now everything went pretty smoothly, Rails, Apache2, Lighttpd, MySQL and PHP4 were all installed without much hassle.

But when I tried to login as root to the MySQL daemon, I kept getting a “Access denied for user ‘root’@’localhost’ (using password: YES)” error message. The odd thing was that I did not set a password for the root account during installation. Some searching on the internet suggested resetting the root password. However, that didn’t work, nor did reinstalling MySQL.

The issue was finally solved after following the suggestion from Abel Lopez from the comments sections of this page in the MySQL documentation. It turns out that the mysql user which owns the processes spawned from MySQL daemon didn’t have enough privileges for the data directory. The workaround was to remove /opt/local/var/db/mysql5/, recreate it and set its ownership via chown mysql:mysql.

After this was done, I re-ran the database install script and sure enough, I was able to login as root and change the password as well.

Have full disclosure for code vulnerabilities

The Rails core team released 1.1.6 of the framework today, a day after 1.1.5 was released. This was to fix a serious vulnerability in the Routes module. The core team has been extremely prompt in publicising the hole and in releasing fixes.

However(you know there had to be one), I take issue with how the first fix release (1.1.5) was handled. It appears that this release did not fully rectify the problem, hence the need for 1.1.6. While DHH revealed the reasons for 1.1.5, he did not detail exactly what was wrong, opting for a security through obscurity approach.

In retrospect, a full disclosure policy would have been a better move. This would have given developers more information in deciding whether to shut down their sites, in view of the implications(data loss/theft et al) of having it compromised.

That said, if you’re running a rails web application in the wild, UPGRADE NOW.

EDIT: mixed up my rails versions, doh!

e27

I had the pleasure of attending e27, which is modelled after Valley un-conferences like FooCamp and BarCamp. The atmosphere was mostly casual although more than a few attendees were in formal getup. A consequence of holding it on a weekday, I suppose. Otherwise, everyone were relaxed and jazzed about being there. Bjorn and Justin were also great hosts who were simply bursting with enthusiasm. The scheduled demos were very interesting but the Q&A sessions sometimes rambled on without a clear purpose.

In all, I’ll say that this was a success and I will be looking forward to the next one.

Automatically creating files for file_column models in Fixtures

I wanted to be able to have my testing regime automatically create files associated with model fixtures defined in the fixture files. Coincidentally, I’m using Sebastien Kanthak’s file_column plugin for managing files in models.

After looking at the Fixture class and the FileColumn module, I realised that I needed a way to store the model attribute(s) that were passed in the call to file_column. What FileColumn did was create the methods according to the attribute passed into the method but the attribute itself is not actually stored anywhere.

Time to extend FileColumn:

[ruby]require File.join(RAILS_ROOT, ‘vendor’, ‘plugins’, ‘file_column’, ‘lib’, ‘file_column’)

module FileColumn
module ClassMethods
@@file_column_attributes = {}
alias :aliased_file_column :file_column
def file_column(attr, options = {})
aliased_file_column(attr, options)
klass = self.name.constantize
@@file_column_attributes[klass] ||= []
@@file_column_attributes[klass] << attr
end

def file_attributes
@@file_column_attributes[self.name.constantize]
end
end
end[/ruby]

As the name implies, the methods in ClassMethods are class methods on the including class. So in order to store the file_column attributes, we need a class variable, @@file_column_attributes. This variable is a hash with the model class name as the key and an array containing the attributes.I'll also implement a class method so that I'll be able to do Model.file_attributes and know what attributes have been file_columnised.

Next, we'll need to change the way fixtures are populated to the test database. This involves extending the Fixtures class.

We'll do this in test_helper.rb

[ruby]
require 'fileutils'
class Fixtures
SAMPLE_FILE = File.join(Test::Unit::TestCase.fixture_path, 'files', 'sample.pdf')
include FileUtils

alias :original_insert_fixtures :insert_fixtures

def insert_fixtures
original_insert_fixtures
create_files(@class_name, values) unless @class_name.constantize.file_attributes.nil?
rescue NameError
# workaround for HABTM fixtures
end

def create_files(klass, values)
model_dir = File.join(Test::Unit::TestCase.fixture_path, 'media', 'uploads', klass.downcase)
values.each do |fixture|
klass.constantize.file_attributes.each do |attr|
create_file(fixture, attr, model_dir) unless attr.nil?
end
end
def create_file(fixture, attribute, parent_dir)
attribute = attr.to_s
dest_dir = mkpath(File.join(parent_dir, attribute, fixture['id'].to_s))
file_dest = File.join(dest_dir, fixture[attribute])
cp(SAMPLE_FILE, file_dest) unless fixture[attribute].nil? or File.exists?(file_dest)
end
end
[/ruby]

The extended insert_fixtures method first invokes original_insert_fixtures, then checks whether the model has any file_column attributes. If it does, the files defined in the fixtures will be created if they don't already exist.

EDIT: Running functional tests with rake test:functionals fails miserably even though running the tests individually is just fine.