RunTime Blog

Our very own musings (mostly) about Ruby, Rails and Software Development in general

...a JRuby project that combined Java and Rails. Everything went smoothly and according to schedule, and the quality of the deliverable was high. Morten M. Christensen, CEO, 41concepts (Denmark)

Metaprogramming in Ruby - Part I

Here at Runtime Revolution we love Ruby. We love it for several reasons but one of the traits we most cherish is it's metaprogramming capabilities.

In this series of posts, I'll introduce and explain the concept, make a tour of Ruby's metaprogramming infrastructure and explore some common metaprogramming techniques.

What is metaprogramming?

Metaprogramming is the process by which a program manipulates other programs or itself.

Although the concept may be somehow strange to some developers, metaprogramming is something that has presence on the daily life of a software maker. Maybe some examples of every day typical metaprogramming products will shed some light on it's ubiquity:

Lexical analyzer generators

Lexical analysis consists on translating an expression into a list of it's representative tokens. This is the first stage of interpreting or compiling your favorite programming language source code files.

Lexical analyzer generators are applications that given a set of scanning rules, produce a lexer. Hence they are software applications that write other software applications given that a lexical analyzer generator output is a lexer (a.k.a. Scanner or Tokenizer).

Interpreters/Compilers

Interpreters or compilers are canonical examples of the use of metaprogramming. Both receive a representation of a set of instructions and produce some other representation of the same set of instruction but as another software program. In the case of MRI 1.8.7 or lesser, your Ruby source code will turn out to be interpreted and executed by C code (for 1.9 and beyond, things are slightly different). In the case of JRuby, a Ruby implementation for the Java Virtual Machine, at a given point, your code will run as JVM's byte code and possibly just-in-time compiled to machine code. In both cases we have a program that writes (manipulates) another program.

C++ Templates

C++ templates are probably the most typical example of a subset of metaprogramming unsurprisingly called Template metaprogramming. With C++ templates you write code which in turn will be pre-processed to standard C/C++ source code that will be merged with the rest of the code base and subject to the regular compilation step.

Most software written in "dynamic" languages

Most software written in Ruby, Lisp, Python, Smalltalk, and other languages able to alter the program at runtime. This is usually referred as dynamic metaprogramming.

This is true even for the most unaware of developers, since in these environments, even if the developers don't actively use metaprogramming features, most libraries and even core language constructs use them. For instance, in the case of Ruby, metaprogramming is present at various levels of the Ruby ecosystem. It's is massively employed in Ruby on Rails but also in Ruby itself. e.g. the attr_acessor method.

What is dynamic metaprogramming?

Dynamic metaprogramming is the process by which a program manipulates itself at runtime.

The first three examples above illustrate a metaprogramming pattern where the resulting program lives on a separated runtime of it's producer. A compiled program, for instance, not only runs on a different runtime but typically is a loose representation of your original implementation, where many of your origin language concepts and constructs (e.g.: scopes, classes, etc.) don't even exist.

This is not the case with dynamic metaprogramming. With dynamic metaprogramming you manipulate your program, on the fly, within the same context. We may think of languages with support for dynamic metaprogramming as languages that have a built-in API to themselves.

Why dynamic metaprogramming?

Why dynamic metaprogramming? It's not that we don't already have our share of concepts, methodologies, paradigms and techniques that we need to throw one more for software developers to wrap their heads around. What's the gain?

Provide an elegant and powerful way to handle cross application concerns

Since using metaprogramming features allows you to write code that deeply interacts and changes code yet to be written by another developer, it's a very convenient way to address cross-application concerns and develop in a very generic fashion. This in turn is one of the features needed to be able to create high level libraries that correctly interact with application code. At this point you're able to write libraries that handle functional (instead of technical) issues, resulting in effective code reuse at a very high level.

In the end, this allows for some very efficient transfer of complexity from application developer to framework/library developer, benefiting an entire language ecosystem.

Dynamic code generation is superior to static code generation

UI Generation libraries provide a good showcase for the above statement. With more static languages (e.g. C/C++) every take on UI generation was made by generating code. This would often result in marginal gains of productivity simply because that customization of generated code would invalidate recreation of the UI code. Regenerating would mean crushing any customization made before hand. This is not an issue with dynamic code generation.

Easy internal DSL development

Dynamic metaprogramming features are very useful for creating internal DSLs. That means you can create a very high level of abstraction for a particular problem, high enough for someone with a functional knowledge of a given domain to be able to use your (mini) language without general programming knowledge. Even if creating a DSL is not desirable, metaprogramming features, when used properly, are useful in creating very high level APIs.

Ruby's metaprogramming infrastructure

Ruby has a very complete metaprogramming infrastructure, some of the most relevant components are:

Strong introspective abilities

Type introspection is the ability that a program has to examine it's own types.
it enables the creation of further metaprogramming infrastructure. Ruby's introspection is deeply embedded in it's object model , thus creating a very intuitive API.

Runtime creation of core language constructs

In Ruby, you can create most of the core language constructs at runtime. This means you can create new classes and methods (or even remove existing methods). Changing existing classes is also possible (this is usually refered to as Open Classes), even core classes.

Other features useful for metaprogramming

Ruby has several other features useful for metaprogramming, such as Metaclasses, Hook Methods (method_missing, included, inherited, etc. ), Closures and methods as first level citizens, and many others.

We will delve into this topics on the next series of this post which will have a much more hands-on approach. I promise :).

Meanwhile, if you want to learn a bit more about metaprogramming in Ruby, check out the excellent Metaprogramming Ruby.