This is a rehash of something I've written before, but I keep returning to this issue.

I wrote this in response to the ongoing post about PHP. This is in follow up to the pro-PHP rant closing the discussion.

Harry Fuecks is on target when he points to the execution model, but I think I can get more specific about two ways that PHP's execution model has made it successful:

• Cheap Hosting (a short point)
  
• It's Still a Scripting Language (a longer point)

Cheap Hosting

Fuecks points out how the execution model eliminates hard to debug user land memory leaks. He also points out how this makes PHP servers easier to administer.

I'd like to also point out that execution model has no dependency on a particular web server scaling model. This allows PHP to be ported to almost every existing web server.

By making PHP an easy business to manage, and by eliminating server/os specific niches in the hosting market, PHP hosting has become a cheap commodity. It's basically free.

It's Still a Scripting Language

Moving on to the rant. Basically, I'd like to raise awareness of this single point:

Scope management is as important as dynamic typing in the development of general purpose scripting languages.

Because I would really like to ruby online, but the architecture insists that I write fire breathing applications rather than the simple pet scripts that I prefer.

Aside from dynamic typing, I believe that the most distinctive feature of general purpose scripting languages (perl, python, ruby, etc) is the support for writing programs of a limited scope. I can write a script confident that it will only affect the problem at hand. For all inputs? Who cares, it only needs to work for this input.

Using ruby as my most familiar example, in a basic command line script I have a range of scopes available:

+ Create a ruby file (script)
+ Dynamically decide how to setup globals (per invocation)
+ Require a library in my home directory (user)
+ Use some standard libraries (ruby)
+ Install some gems (machine)

Just by installing ruby I can write little scripts focused on my problem while leveraging shared code from the larger community. And I am in control of when I pull in successively "more" global code.

But when I go online, I lose some of this control. The most common means of scaling general purpose scripting languages is to simply span the interpreter across multiple requests (with various tricks). I believe that this lazy route causes a loss of "scripting language" features. Continuing to use ruby as an example:

CGI: Create an interpreter per invocation

CGI has the same available scopes as the home directory, except that even a trivial web site will cause more invocations than one would see with a commandline script. It doesn't matter that I still have all these scopes available because the server dies of load.

FastCGI: Create 1 to 10 interpreters per script, reuse the interpreters

- limit on the number of scripts
All those interpreter instances will consume between 5 and 10 megabytes of memory. That overhead can add up, and is enough to discourage me from creating many scripts. Unfortunately, this loses a nice feature of scripting -- the ease of forking.

- per invocation scope lost
I get to worry about memory links since the interpreters are re-used. Another feature of scripting lost. Plus, I can't do any dependency injection tricks b/c I need to be careful about setting globals per invocation. This is 'free' in PHP and in commandline ruby, but for ruby online it's another feature of scripting lost.

Mod Ruby: Create 1 interpreter per web server process

- script scope lost
- per invocation scope lost
- user scope lost

Mod Ruby is quite bloody. Since the same interpreter is used across any requests to the web server, you lose scopes smaller than the machine. All development happens at the scope as the web server, which is usually on the machine level. You have the memory leak / globals issues of FastCGI made worse by the fact that now unrelated scripts will interfere with each other.

This doesn't sound like scripting at all to me :-)

Conclusion

As far as I'm concerned, ruby is not a web scripting language. Neither is perl, python, or tcl. Once used online, scaling these languages turns them into a variant of SmallTalk, i.e. a dynamically typed systems language. PHP is the only open source scripting language of which I am aware.

It's worth noting that Coldfusion (another horrid language with in-comprehensible success) does a good job of separating requests from each other. While there is a less than obvious auto-include of application.cfm files, requests are independent.

I've uploaded <data> support to plist.rb. Special thanks goes to Mat Schaffer, who did the hard work of finding a test case. <data> elements are returned as Tempfiles, and can be replaced with an IO or StringIO:

# reading <data> elements
data = Plist::parse_xml("example.plist");
data['image'].read

# changing <data> elements
bin = File.open("example.jpg"){ |f| f.read }
data['image'] = StringIO.new( bin )
data.save_plist("example_data-v2.plist")

The easiest way to install is to "gem install plist"

Flying over the Atlantic, I wondered about the reliability of the flight progress indicators. It's so hard to integrate software. How did the engineers pass the location of the plane to one critical system (autopilot) and one trivial system (cabin TVs) while ensuring that a short or other problem with the cabin TVs didn't knock out the autopilot. My memory of the East Coast blackout lingers.

My father-in-law is a process engineer and amateaur pilot. After landing, I asked him how the engineers did it. He explained that there most likely is just a second GPS for the cabin TVs. In fact, the electrical system for the cabin is probably completely separate from the avionics. The reward of sharing one component (the GPS) isn't worth managing the additional risk of merging the circuits.

Consider the different areas of a website, and the different divisions of a company. It isn't always worth allowing the marketing department to take risks that affect fulfillment. Especially if the code between business units is straightforward:

def fullname; firstname + ' ' + lastname; end

Having 2 copies of the above code doesn't hurt anyone.

Furthermore, given:

• one web domain is one business
• the one web domain is handled by one web application
• that business employs a full time programmer
• on some regular schedule, that programmer releases code to production
• it is impossible to catch all bugs
  

On some regular schedule, the entire website is at risk. It frightens me -- the entire business as a single (failure prone) application.

On the other hand, if these are given:

• the website is split into 5 web applications
• on some regular schedule one of the web applications is updated

Then only 1/5 of the website is at risk at a release point. Even better from a risk management perspective, given this environment:

• 500 unlinked php/coldfusion/asp scripts

Each update only puts one small fraction of the site at risk. My point is not that the 500 scripts is the optimum solution, but that linking code also links risk.

In my view, linking code is a bet that the benefits will outweigh the risks. There is nothing wrong with taking risks, but sticking to a simple rule like "Don't Repeat Yourself" can obscure the bets being made.

It is worth knowing when you are standing on the shoulders of giants, and when you are standing on thin ice.

When discussing whether a web technology "scales", usually the conversation alternates between two basic questions:

1. Does the implementation perform acceptably? Does it continue to perform acceptably as load is increased in orders of magnatude?
2. Can the implementation be maintained and enhanced?

To show that a technology "scales", one must illustrate how both questions can be answered at the same time without sacrificing one for the other.

The simplest way to run Ruby from a web server to use CGI. Unfortunately, with CGI applications each additional request loads a new interpreter with a new instance of the program. This leads to the notoriously bad performance of CGI under increased load.

Ruby has several ways of serving web pages which offer better performance:

• mod_ruby: embeds the interpreter into the Apache server
• fastcgi: using the FastCGI protocol, scripts run in a loop and hang around to serve multiple requests
• webrick / mongrel: 100% ruby web servers. Often these are integrated with a major web server like Apache or IIS via proxying.

Note that each of these implementation persists an interpreter across requests:

• mod_ruby: one interpreter per Apache process
• fastcgi: one pool (1-10) of interpreters per script
• webrick / mongrel: one interpreter per "site", but different proxying strategies can lead to the interpreter being shared in various ways.
  

When the same interpreter is used across different requests, there are important consequences to the programmer:

• require 'lib'
  The first call is processed, and the rest are skipped. If you are dynamically adjusting your load path, the question becomes: Which 'lib' was loaded?
  
  
• overloading core methods
  Can one be sure that adjustments to the Array class are ok for every instance of an Array on the site?
  
  
• global variables leading to memory leaks
  If it is possible for variables to be referenced between requests, these references can prevent garbage collection and lead to memory leaks

These concerns illustrate that when scaling ruby, your scripts have become an Application [1]. They also illustrate the difficulties that ISPs face when hosting Ruby, since for security reasons an ISP must force "Share Nothing" behaviour between users.

Scaling Ruby to provide performance forces a change in development mentality. This transition from writing scripts to writing applications increases the cost of maintenance. For this reason, I do not think that Ruby scales on the Web [2].

Luckily, I consider this to be a failure of the current Ruby web implementation, not a fact of the web development problem space. We have examples which can be followed.

The commercial web scripting languages (ASP and ColdFusion) allow casual "scripts" to perform well. A bit more useful example, PHP has implemented a web scripting language which performs acceptably without requiring developers to go into "Application Mode."

• All caching is performed by the implementation under the covers. It is impossible for hosted code to share anything between requests.
• Every request is given a fresh and identical operating environment, regardless of webserver.

It's worth dwelling on PHP for a moment longer. In PHP, it is possible for a novice to deploy multiple large-scale multi-developer projects (PHP-BB, WordPress, Drupal) on the same $4/month shared hosting environment.

It is imperative for the future of Ruby that we recognize how the implementation of PHP has lead to dominance of the Web scripting market [3]. Ruby must outgrow its Perl roots and discard the notion that large-scale web programming requires the "sobriety" of Application development.


Footnotes

1. Note that the change in developer mentality has also been observed in mod_perl development.
2. There's another insiduous property to Ruby's implementation. Not only does the implementation push developers to think in Application Mode, it also pushes developers to choose one Application Framework per web server. The forthcoming arguments of Rails vs XXXX vs XXXX will be a waste of time.
3. Even though the (easy) comparison to Java is a seductive angle, it is important to note that every other environment also shines when compared to Java. Tcl, Python, and now even mod_perl are niche markets. Ruby will remain a popular novelty unless it compares itself to the real competition.
   

Swish-e is a command-line full text indexer similar to ht://Dig. Swish-e sports a brilliant hack. When indexing, one can ask Swish-e to index the output of a command:

swish-e -S prog -i ./output_documents.rb

The output_documents.rb dumps out a series of html documents:

#!/usr/bin/ruby

require 'dbi'
require 'pp'

dbh = DBI.connect("dbi:Mysql:test:localhost", "user", "pass")
# get server version string and display it
artbase = dbh.select_all("SELECT * from objects")

artbase.each do |object|
  path = '/object/' + object['object_id'].to_s
  mtime = object['modified'].to_time.to_i
  html = <<HTML
<html>
<title>#{object['title']}</title>
<meta name="author" value="#{object['from_string']}">
<body>
#{object['body']}
</body>
</html>
HTML
  print "Path-Name: #{path}\r\n"
  print "Last-Mtime: #{mtime}\r\n"
  print "Content-Length: #{html.length}\r\n"
  print "\r\n"
  print html
end

Swish-e is FAST, in every way that matters:

* Installation is simple -- it's just a few commands
* Indexing and searching is speedy
* Integration is a breeze, since you are just executing commands.

Discovering Swish-e is like witnessing the birth of Athena, fully grown and ready for battle. I highly recommend it.

There is an important security note for use with websites. I'll post more when I work out the Ruby equivalent of the recommended Perl techniques.

In the other half of my day, I am a painter. For about half of a year, I've been capturing snapshots of my work on some digital prints. I've been just showing these running down a page, but I haven't been happy with it:

http://hexane.org/slider/artwork-old.html

Last night I pulled this work together using dHTML:

http://hexane.org/slider/artwork.html

The programming is nothing too exciting: script.aculo.us is cool, browsers are still annoying, but all it takes is a few hours of fiddling.

The insight for me was: given interesting enough data (meat), all you need is an evening of whiz-bang javascript (gravy) to put together something nifty.

Of course, if you prefer the running-down-the-page version, let me know. That way, I can go to bed earlier tomorrow night :-)

Just released a library for parsing iphoto databases:

gem install plist
gem install iphoto2

iphoto = IPhoto2.new # path to AlbumData.xml is optional
iphoto.albums.each do |album|
  album.each do |image|
    puts image.path
  end
end

There's more, but I'm tired now.

Update: oylenshpeegul pointed out a bug in the comments, I've put out another gem (iphoto2-1.0.1) to the problem with the Album#each. And he is correct, there is an Album#images accessor.

To help with my AlbumData.xml parsing, I've thrown together a plist parser for Mac OS X plist.xml files. Api documentation: Plist::parse_xml( plist_xml_filename )

~/Pictures/iPhoto Library/

2006/01/07/fallow_keep.png.450x450.2005-12-04.jpg

original image

2006/01/07/Thumbs/7.jpg

a thumbnail

AlbumData.xml

plist file with all the goodies

Dir.data

binary

iPhoto.db

binary

iPhoto.ipspot

text, perhaps for spotlight indexing?

iPhotoLock.data

blank at the moment

Library.iPhoto

binary

Thumb32.data

binary

Thumb64.data

binary

ThumbJPG.data

binary

I kept the library pretty simple, to show the structure:

{
    "Application Version" = "5.0.4 (263)"; 
    "Archive Path" = "/Users/username/Pictures/iPhoto Library"; 
    "List of Albums" = (
        {
            AlbumId = 999000; 
            AlbumName = Library; 
            KeyList = (7); 
            Master = 1; 
            PhotoCount = 1; 
            PlayMusic = YES; 
            RepeatSlideShow = YES; 
            SecondsPerSlide = 3; 
            SlideShowUseTitles = 0; 
            SongPath = ""; 
            TransitionDirection = 0; 
            TransitionName = Dissolve; 
            TransitionSpeed = 1; 
        }, 
        {
            "Album Type" = "Special Roll"; 
            AlbumId = 999001; 
            AlbumName = "Last Roll"; 
            "Filter Mode" = All; 
            Filters = ({Count = 1; Operation = "In Last"; Type = Roll; }); 
            KeyList = (7); 
            PhotoCount = 1; 
            PlayMusic = YES; 
            RepeatSlideShow = YES; 
            SecondsPerSlide = 3; 
            SlideShowUseTitles = 0; 
            SongPath = ""; 
            TransitionDirection = 0; 
            TransitionName = Dissolve; 
            TransitionSpeed = 1; 
        }, 
        {
            "Album Type" = "Special Month"; 
            AlbumId = 999002; 
            AlbumName = "Last 12 Months"; 
            "Filter Mode" = All; 
            Filters = (); 
            KeyList = (7); 
            PhotoCount = 1; 
            PlayMusic = YES; 
            RepeatSlideShow = YES; 
            SecondsPerSlide = 3; 
            SlideShowUseTitles = 0; 
            SongPath = ""; 
            TransitionDirection = 0; 
            TransitionName = Dissolve; 
            TransitionSpeed = 1; 
        }, 
        {
            "Album Type" = Regular; 
            AlbumId = 9; 
            AlbumName = "An Album"; 
            KeyList = (7); 
            PhotoCount = 1; 
            PlayMusic = YES; 
            RepeatSlideShow = YES; 
            SecondsPerSlide = 3; 
            SlideShowUseTitles = 0; 
            SongPath = ""; 
            TransitionDirection = 0; 
            TransitionName = Dissolve; 
            TransitionSpeed = 1; 
        }
    ); 
    "List of Keywords" = {}; 
    "List of Rolls" = (
        {
            "Album Type" = Regular; 
            AlbumId = 6; 
            AlbumName = "Roll 1"; 
            KeyList = (7); 
            Parent = 999000; 
            PhotoCount = 1; 
        }
    ); 
    "Major Version" = 2; 
    "Master Image List" = {
        7 = {
            "Aspect Ratio" = 1; 
            Caption = "fallow_keep.png.450x450.2005-12-04"; 
            Comment = ""; 
            DateAsTimerInterval = 158341389; 
            ImagePath = "/Users/username/Pictures/iPhoto Library/2006/01/07/fallow_keep.png.450x450.2005-12-04.jpg"; 
            MediaType = Image; 
            MetaModDateAsTimerInterval = 158341439.728129; 
            ModDateAsTimerInterval = 158341389; 
            Rating = 0; 
            Roll = 6; 
            ThumbPath = "/Users/username/Pictures/iPhoto Library/2006/01/07/Thumbs/7.jpg"; 
        }; 
    }; 
    "Minor Version" = 0; 
}

Just a few notes from working on supporting FastCGI on windows:

• fcgi.rb is not compatible with Windows.
  • Simple incompatiblities include trapping undefined symbols on Windows.
    
  • $stdin is not always a socket on fastcgi on windows, so wrapping $stdin.fileno doesn't work
    
  • libfcgi has OS dependent code to handle various windows cases
    
  
  
• compiling libfcgi for windows is difficult
  • Makefiles have a few syntax errors with nmake
    
  • assembling a free compiler enviroment on windows is a pain
    • MS VC Toolkit
      
    • MS Platform SDK
      
    • MS.Net SDK
      
    • ... or buy MS Visual Studio
      
    
    
  
  
• isapi_fastcgi has alternate version of libfcgi, but with Visual studio build files (.dsp)
  

So I guess I need to get Visual Studio to properly support Fastcgi on windows.

Ruby continuations have been described as "going back in time" in your program, but that's not really true. You go back into the stack, but all your state remains the same. This ruby snippet:

state = "label 1"
continuation = callcc { |c| c }
puts "state: #{state}"
state = "label 2"
continuation.call if continuation

Will produce the following output:

state: label 1
state: label 2

Note that the state variable keeps it's value. This is how you can do useful programming with continuations, even though you've jumped "back in time."

The next wierd thing, is that we don't get stuck in a loop. Notice "if continuation"? The second time we try to call the continuation, it is nil.

If you want, you can make a loop by doing the assignment inside the callcc block:

state = "label 1"
continuation = nil
callcc { |c| continuation = c }
puts "state: #{state}"
state = "label 2"
continuation.call if continuation

That will give you this output:

state: label 1
state: label 2
state: label 2
state: label 2
... (ad nauseam)


Chad Fowler just linked to an article describing local maximums and career optimization. I guess Point C is the same as 3. ????.

Each time I sit down and think about Ruby vs other web environments, I see so much room for improvement. I thought the scary stuff was behind me, and then I started thinking about database support.

The next dip will be in sorting out the nitty gritty of database support. I think the main problem with DBI is driver support -- that'll teach me C for sure! Sorting out easy to use binary installers is also a pain.

Btw, Kirk Haines, developer of IOWA, pointed out that IOWA already has developed a Drb connection pool.

I've been working out a contradiction in web apps:

• it's nice for each page to be independent, without having to worry about changes from other enviroments
  
• it's nice to have just a few cached database connections
  

I can use continuations to make repeated ruby requests independent, but the problem is that I want a few objects to persist. Implementing something that will roll back modifications to Kernel while preserving a few database connections seemed daunting. Until I realized ... run the connections in a separate process and communicate using drb.

Update: Ara Howard has released a postgres connection pool

You can get MVC action by combining two features of ruby-web. Before loading a page, ruby-web looks up the directory tree for an "application.rb" file. Then, in the application.rb file, you can define a Web::filter [1].

Let's use a basic hello.world.rb as an example:

#!/usr/bin/env ruby-web
puts "Hello World"

When you visit this cgi script, you'll get "Hello World". If you add this application.rb in the same directory as hello.world.rb:

Web::filter do |content|
  "<h1>#{content}</h1>"
end

Visit the script again, and notice that Hello World is a bit louder. You can also have multiple chained filters:

Web::filter do |content|
  "<h1>#{content}</h1>"
end

Web::filter do |content|
  content.gsub(/Hello World/, "Hallo Welt")
end

Take that, Servlet API!


A few notes:

1. If you are running ruby-web 1.0.0, Web::ob_start is a more cryptic name for the exact same function. If you are running php, ob_start is also your friend :-)

2. I'm sorry that ruby-web scripts still run as basic cgi scripts. Alas, I have not yet written enough SAPI modules. One day I'll get them all done.

This site was dedicated to the NARF web libraries. These have been moved to http://www.ruby-web.org. From now on, NARF will be dedicated to an attitude towards projects. That what Ruby needs isn't big ideas, but problem solving code.

In other words, this site will be dedicated to the attitude that generated ruby-web and raainstall. I will celebrate the can do, pragmattic, software engineering mentality that has made projects like Perl and PHP so friendly.

I will continue to post updates on the status of ruby-web, but I will also feature useful projects like the Nullsoft Installer System, Docbook, and whatever else is friendly and productive.

I hope to follow the example of these great projects. Cue the fanfare, courtesy of http://www.frazmtn.com/~punstr/midi/.