every domain-specific framework, there is a language
crying to get out.
Jay Peckish II
number of forces shape the way
software evolves. One is a desire to
make programs as reusable as possible.
Another is to push configuration decisions out into the data. Yet another is to push such decisions out
onto the users. Still another is
to defer such these decisions until runtime.
Data themselves become more universal and reusable when they
are accompanied by descriptions of themselves that let other programs make
sense of them. They can become even
more independent when they are accompanied in their travels by code.
The patterns in
begin to chronicle how domain specific languages emerge as programs
evolve. A program may begin simply,
performing but a single task. Later,
programmers may broaden its utility by adding options and parameters. When more configuration information is
needed, separate configuration files may emerge. As these become more complex, entries in these files may be
connected to entities in the program using properties, dynamic variables, and
dialogs. Simple values may not
suffice. Once properties and dynamic
values are present, simple parsers and expression analyzers often are added to
the system. This, in turn creates a
temptation to add control structures, assignment, and looping facilities to the
functional vocabulary provided by the expression analyzer. These can flower into full-blown scripting
After a while, the domain or business objects come to
constitute a program of sorts, which can be dynamically constructed and
manipulated by users themselves. During
this evolutionary process, descriptions of the data, such as maps of the
layouts of data objects, and references to methods or code, are needed to
permit these heretofore anonymous capabilities to be accessible during runtime.
These descriptions allow these objects to be composed, edited, stored,
imported, exported, and (these are programs, after all) debugged.
As this evolutionary process unfolds, and the architecture
of a system matures, knowledge about the domain becomes embodied more and more
by the relationships among the objects
that model the domain, and less and less by logic hardwired into the code. Objects in such an ACTIVE OBJECT
are subject to runtime configuration and manipulation like any other data. Changes to this runtime constellation of
objects constitute changes to the model, and to the operations that traverse or
Data that describe other data, rather than aspects of the
application domain itself, are called metadata. Naturally, these layout and code
descriptions should be objects too.
Hence, metadata have metadata as well.
A successful application inevitably draws a crowd. A host of users on a hosts of hosts will
want to use such a program, and the data that go with it. It is important that data produced by one
copy of the program be usable by other users at other sites. Such data might reside in a shared or
distributed repository such as a database or persistent object base. They might also migrate across a network,
via wires, satellites, fibers, radio waves, and even diskettes or tapes.
It is important, too, that these data be accessible not only
from copies of the applications that spawned them. Other programs must be able to deal with them as well. When such data are mere "punch card
images", or undifferentiated byte streams, this is hard to do. However, when data are escorted by machine
readable descriptions of what they mean, they become welcome in a wider range
of processing venues.
Our story then, is about how data earn their wings. It chronicles the forces that drive data to
become more general. It describes their
ascent from digits on punch cards, to lines on data files, and bytes in
through structures, and on through their marriage to
behaviors, which begot objects. It
continues as the need to describe these objects incubates self-descriptions,
which themselves are cast as objects, which, in turn, allow objects to aspire
to escape the processes and images in which they were trapped, and roam
unencumbered across the network.
The drive to become more general begins modestly. A simple application may acquire command
line switches and parameters, to allow its behavior to vary, or permit
additional input streams to be specified.
As a program becomes yet more general, additional configuration
information may be needed. This information
may complex, and may even be provided interactively, by end users. Simple, textual interfaces may yield to
graphical user interfaces, which themselves may grow more powerful, and, alas
As an object-oriented application evolves, the elements of a
object-oriented framework emerge. Where
raw, undifferentiated, white-box code once was, dynamically pluggable black-box
components begin to appear. Internal
structure, which was once haphazard, becomes better differentiated, and more
As such a framework evolves, the these elements themselves,
together with the protocols and interfaces they expose, come to constitute a
domain specific language for the framework's target domain.
Often, something else happens as well. The configuration user interface and tools
grow more powerful too, so as to expose more and more flexibility and power to
the users. At first, simple parameters
are exposed. Later, expressions and
simple logical rules may be proffered.
Finally, control structures might emerge, and the full power of this
emerging language is exposed to the user.
Users may be offered existing behaviors, or new behaviors might be added
using scripts which might be interpreted, or even compiled at runtime. Editors emerge that allow users to directly
manipulate the objects that constitute their "programs".
This story might have a familiar ring to those readers who
have followed the research done over the years into reflection and metalevel
architectures. Of course, the
reflection literature has earned it's recondite reputation the hard way (that
is, through unrepentant abstruseness
.) Our tale might be seen as an attempt to
render their Finnigan's Wake as, if
not a Mother Goose Tale, at least a
trip Through the Looking Glass.
The patterns in this paper are part of a larger pattern
language that we are writing. We
currently envision a language that will include the following patterns.
The patterns included in this
PLoP '98 version of this work are shown in bold:
METADATA patterns in this collection
can be broken down into the following categories:
Patterns that arise from pushing
decisions out onto the user:
Patterns that arise as a domain
specific languages emerges:
10. SMART VARIABLES
11. SCHEMA / DESCRIPTOR
12. ACTIVE OBJECT-MODEL
VALUE HOLDER / SMART VALUES
Patterns that become relevant as data become "self aware" (or more reflective)
24. SYNTHETIC CODE
25. CODE AS DATA
26. CAUSAL CONNECTION
A variety of forces impinge upon evolving systems. Some of them pervade the pattern
language below, and are enumerated here to avoid duplication:
Portability: When an artifact works with a variety of
applications, on a variety of platforms, it is more likely to be reused.
Efficiency: Highly dynamic systems can be inimical to
efficiency. However, efficiency is
often a false idol. For instance, the
cost of referencing an object in a remote database may be several orders of
magnitude more expensive than accessing a local object, and such overhead may
overwhelm secondary concerns, such as the cost of accessors vs. direct variable
Complexity: Complex data structures and code are hard to debug
and comprehend. Alas, many programmers
are better at creating complexity than simplicity.
Dynamism: Interactive programming environments, visual builders
and debuggers, and distributed applications all benefit from a more dynamic
approach to software system architecture.
Dynamism can be dangerous, though. More dynamic systems can be harder to debug, maintain, and
understand. One wouldn't let a child
learn to ride a bicycle on a busy highway.
Resources: Dynamic strategies can be costly in terms of space,
processing time, secondary storage, etc.
Safety: Dynamic strategies allow users to circumvent and
undermine compile-time safeguards.
Flexibility: A program should be versatile, and usable in a
variety of contexts. This, in turn
Reusability: A versatile, flexible application, or, for that
matter, a code-level artifact, should be as reusable as possible. The reuse of such code avoids duplicated
effort, eases the learning and comprehension burden of new programmers, and
makes maintenance easier, since multiple, redundant copies of essentially the
same code need not be maintained.
Adaptability: It is essential that an artifact be flexible
enough so as to confront and address changing requirements. We distinguish several "shades" of
Maintainability: It is important that an artifact be
maintainable enough to as to confront and address changing requirements. Code that can't be worked on will lapse into
Tailorability: One size does not fit all. Often, an artifact will not fit the needs of
a particular user "off the rack", but can be tailored to do so
when certain "alterations" can be made.
Customizability: Just an artifact can be tailored to a particular
user or users, it can be customized to adapt it better for a particular task. This may seem at first to be a lot like
tailorability, but we find that distinguishing between forces for change than
emanate from individual users and those that arise from taking on different
Pushing Complexity into the Data: When complexity is pushed
into the data, it can be coped with dynamically, at runtime. Configuration information can travel with
the data, rather than being locked up in explicit code.
Pushing Configuration Decisions out onto the User: As a
framework evolves, more and more configuration decisions are pushed out onto
the user. Users become programmers of
sorts. The trick, of course, is not to
force them to be general purpose programmers.
They don’t have the training for this, and would fear that their social
lives would be ruined. And, real
programmers would be out of jobs.
Autonomy/Mobility: Once behavior and data, together with their
descriptions, are liberated from application code, they can travel
independently of these applications, and be used in a wider range of programs,
on a wider range of platforms.
Comprehensibility: Metadata helps to document its
associated data. Indeed, data files
with metadata in them were often referred to as “self-documenting” data files
during the ‘70s.
also known as
v v v
How do you allow your program to compute more than
For the most part, a program that computes the same
result every time is of little use.
arrange for data to be able to be read in and written out of your program.
The distinction between program and data is so
familiar that it hardly needs to be made.
Still, it will become useful when we discuss metadata and self-modifying
code, so we begin our story here.
Data are bits.
Bits, in and of themselves, are meaningless. Data, therefore, are not merely disembodied bits. Data are about something. Data represent something. Bits, whether parceled into masks, bytes,
words, arrays, or structures, are symbols, that represent relevant notion drawn
from the application domain from whence these notions came.
Random bits are about nothing. Baring a vanishing small,
monkey-at-a-typewriter fluke, a processor almost certainly can't execute very
many of them as non-trivial instructions.
They will make little sense when interpreted as complex data, but can
(to the consternation of programmers tracking down uninitialized memory
problems) masquerade as simple data on occasion. On those rare occasions where "random bits" are used to
simulate stochastic processes, they are
data. Of course, such bits must be
chosen carefully, since generating genuine randomness a chore to which
computers are particularly ill-suited.
Traditionally, data have come in two fundamental
guises, text and binary. Text, in turn,
was strung together as strings and streams.
Text is interpreted in terms of standard character sets, and has
traditionally carved memory into 5, 6, 7, 8, 16 or 32-bit chunks. There are fewer such restrictions with
One good thing about binary data is that if you can
get back the same bits you had the last time you ran, you can pick up where you
left off. This may seem obvious, but it
is actually an important principle. As
long as data are read back in the same order in which they were written out, it
is possible to get the state of memory back to the way it was when the data
were written in the first place. Every
beginning programmer learns this "trick", and it is the cornerstone
of truly traditional (> nine months) streamed data processing.
Complications such as byte order, word size, and
floating point formats enter the discussion quickly, though. More complex objects require more complex
This approach should have a name. Is it CLASSIC STREAMING, or
SNAPSHOTTING? Is it two classic
patterns? It is an example of a broad
principle from which the MEMENTO pattern is derived.
v v v
also known as
ABOUT THE DATA
v v v
How do you avoid the limitations of fixed-format
Data are about what the program does.
Every program has its expectations as to how it
wants its data laid out. When no
descriptive data is present, this "fixed format" must be adhered to
by any data set that wishes to work with a particular program.
Therefore , arrange
for a description of
the data to accompany your data where ever they go.
A better solution is to let data carry a description of
how they are laid out around with themselves.
Subsequent patterns describe how to do this.
You can embed extra flags, descriptive information,
and other interpretable information in your data stream. For instance, a program that operates on
two-dimensional time series data might break it up into chunks, and precede
each chunk with a count of the number of channels and points in each
chunk. A program that reads such data
would read the metadata before each chunk, and then read the data.
A simpler example is a binary stream broken into
chunks where each chunk is preceded by its size in bytes. Another example is the frequently seen
scheme that prepends a signature word to each chunk, both for initial
identification, and for byte-order identification.
v v v
Metadata can have schemas too.
also known as
How do you allow
to add and remove new attributes on-the-fly?
The following minimal set of operations on
properties will usually be present in some form:
void addProperty(Indicator name, Descriptor descriptor, Object
void removeProperty(Indicator name);
boolean hasProperty(Indicator name);
void setProperty(Indicator name, Object value);
Object getProperty(Indicator name);
The hasProperty() will
either be explictly or implicitly present.
When it is not explictly present, a distinguished value such as Property.ABSENT
might be returned by getProperty() and setProperty() to
indicate the absence of a property, or an exception might be generated.
Some implementations don't provide an explicit addProperty()
operation, and allow the first call to setProperty() to
create a new property instead. This is
often the case when property Attributes
are not present.
Similarly, the removeProperty()
operation can be dispensed with by providing for removal of a property when a
designated value is assigned to it, such as Property.REMOVE . This value, naturally, must be one that need
never be the value of a Property .
One or more of the following additional operations
might be present in some form as well:
Descriptor getDescription(Indicator name);
The role, if any of the Descriptor
objects, will vary depending upon the language and implementation strategy
used. In dynamically typed languages
such as CLOS, Smalltalk, or Self, they may not be present at all. In languages such as C++, Java, and C,
minimal type information is might be used to indicate how different property
value should be downcast. It is also
used by tools such as editors, visual builders and debuggers.
runtime mechanisms for accessing, altering, adding, and removing properties or
attributes at runtime.
Sometimes it is difficult to
trace a pattern back to its origin.
This is not the case with PROPERTIES.
We can be quite definite as to where this idea first arose. Properties first appeared in MIT's early
Lisp systems, and were described in the landmark Lisp 1.5 Programmers Manual [McCarthy et al. 1962].
Every atomic symbol in Lisp 1.5 had a property list. The first time a symbol was encountered, a
property list was created for it. In
Lisp 1.5, property lists began with a special sentinel value (-1). The rest of the list contained the
properties themselves, as indicator/value pairs. These indicators, or property names, were themselves atoms. Some of the indicators used by Lisp 1.5
PNAME The print name of
the atomic symbol for I/O
EXPR An S-expression
defining a function whose name is the atomic symbol on whose property list the
defined by a machine language subroutine
APVAL Permanent value
for the atomic symbol considered as a value
Not only does the notion of PROPERTY have a long history,
but it casts a wide shadow. The name
"property" has been used to describe three distinct intents.
Each of these is described herein as a separate pattern. The intents, and the corresponding patterns
PROPERTY You want to add and
remove attributes on a per-instance basis at runtime
SMART VARIABLE You want to augment the behavior of
variable references and assignment
SCHEMA You want a map of
your variables so that you can enumerate them, manipulate them en masse, and
reference them indirectly, using symbolic names
pattern-hood of PROPERTIES was first suggested by Beck [Beck
also known as
How do you allow programmers to control the effects of references to
provide a means for intercepting references to variables.
also known as
Quite often a developer is burdened with the
problem of dealing with structures within the code that evolve rapidly. They may be dealing with a database that
changes or it might be the GUI. Schemas
provide for a way to interpret at runtime these structures thus allowing one to
not have to worry about these types of changes.
How do you avoid hard-wiring the layouts of structures into your code?
How do you describe the layout of a structure, object, or database row?
the layout of a database or structure will change as an application evolves.
coded values of a structure can perform quickly.
coded values of a structure can be easy to understand and debug.
structures at runtime can provide for dynamic values in a system.
structures can have harsh performance hits and can be hard to debug.
Therefore, make a
schema or map describing your data structures available at runtime.
A schema is a language for describing a
structure. Users don’t have to
understand the Schema, they can just use them as is.
Optimize by compiling out problem areas or by
caching in values.
Schema maps require one more level of indirection
to interpret what the code is actually doing.
This can make for code that can be hard to debug and understand.
Database Schemas, IDL Interfaces, C-Structures,
COBOL records, Classes, SCHEMAS might all be discussed here..
Schemas use Interpreters and Builders.
Editors are usually created for editing
usually end up with a Visual Language for manipulating Schemas.
also known as
dynamic object-model is an object
model that provides “meta” information about itself so that it can be changed and
evolved at runtime. Dynamic object-models
usually arise from domain-specific needs or
frameworks. Ultimately these models could become generic
so that they ca n span
several domains (think of a graphing framework that originated in one domain but
then was abstracted to and enhanced so that it can be
used in for any application s needing
runtime configurable makes it so that tools can be developed that allow decision
makers and administrators to introduce new products and changes to the business at
runtime. This can reduce time-to-market
of new ideas from months to days if not hours .
How do you let your users build programs without “programming”? How do you let your users customize and
change the behavior of what they do at run-time?
Users need the
code working today .
Building Dynamic Objects is hard.
Once built, Dynamic Objects allow for rapid alterations
to your program.
You can program without programming.
and business needs change frequently.
Changing a program to meet new business requirements
slow and require much debugging etc.
need the ability to change what they do
Strategies can be hard.
can be difficult to develop, hard to understand, and hard to maintain.
Therefore, develop a
that can define the objects, their states, the events, and the conditions under
which the objects changes state. Include with this model the ability to have a
pluggable Dynamic Strategy that can be invoked and changed on the fly. Also include editors and other
tools to assist with developing and manipulating the object model.
A system with an
has an explicit object model that it interprets at run-time. If you change the
object model, the system changes its behavior. For example, a lot of workflow
systems have an Dynamic Object-Model.
have states and respond to events by changing state. The Dynamic
Object-Model defines the objects,
their states, the events, and the conditions under which an object changes
state. Suitably privileged people can change this object model "without
programming". Or are they programming after all? Business rules can be stored in an Dynamic Object-Model. This makes it easy to change the way a
company models its business.
Today, a new product
typically requires dedicated software support , which requires new
applications or new application releases. This sometimes takes
several months, which is pointless if the particular product lines or business
rules are short-lived and last for a few months. For example, in the insurance
business, new rules for the way that
rates are calculated change quite frequently.
This could take a few months
before the new application could be deployed and
released to the agents. In fact, by the time you released the
application, new rates may be needed.
This can lead to costly maintenance and
evolution, let alone an application running that never really
meets the needs of the end-user.
are definitely harder
to build than a static object-model. The se usually
evolve out of frameworks. A
good pattern to consider when building these models
is the evolving frameworks pattern language that gives good insight to when and
how to write generic configurable frameworks. If your system is only going to change a
couple of times over its lifetime, then the payoff isn’t
high enough to need this pattern.
However, if your system is going to need to change
frequently, or if there are certain rules within the business
that change quite
often and there is a competitive advantage to
allow these changes to get be released within days
rather than months, then a dynamic object-model should be used.
do you build
dynamic object-models . Well, you use parameterization. Metadata is read from databases and objects
are generated from schema descriptions at runtime. Values are usually being interpreted so parse
trees are usually being generated and evaluated. Really
most of the patterns described are support ing
patterns to building dynamic object-models. Maybe this should be an umbrella pattern.
v v v
: Objectiva Telephone Billing Framework, Hartford UDP Framework, Argo School
System, Caterpillar Financial Modeling Tool , I’m
sure Brian knows of more.
You usually use Builders, Interpreters, Generators,
etc to support this pattern.
You can use type object as part of this.
is used heavily here.
pattern is strongly used in most cases.
can also be helpful with this.
are also used.
State pattern can help with evolving states.
We are grateful to our PLoP '98 shepherd, Neil Harrison, as
well as our PLoP '98 program committee overseer Jens Coldewey, and PLoP '98
program chair Steve Berczuk, for their faith, forbearance, and consul during
this paper's somewhat protracted gestation.
Its inadequacies, such as they are, are solely the responsibility of its
We also owe a debt to the participants at the May Metadata
Workshop at the University of Illinois, whose work, ideas, and insights have
helped us to frame our own
Discussions with Ralph Johnson, Dragos Manolescu, and Dirk
Riehle helped shape our thinking about these issues, and their pattern-hood.
The Timeless Way of Building
Oxford University Press, Oxford, UK,
[Alexander et. al 1977]
C Alexander, S. Ishikawa, and M.
A Pattern Language
Oxford University Press, Oxford, UK,
Smalltalk Best Practice Patterns
sPrentice Hall, Upper Saddle River, NJ, 1997
[Bobrow et. al. 1988]
D. G. Bobrow, L. G. DeMichiel, R. P.
S. E. Keene, G. Kiczales, and D. A.
Common Lisp Object System
X3J13 Document 88-002R
SIGPLAN Notices, Volume 23,
Special Issue, September 1988
[Bobrow & Kiczales 1988]
Daniel G. Bobrow and Gregor Kiczales
The Common Lisp Object System
Metaobject Kernel -- A Status Report
Proceedings of the 1988 Conference on
and Functional Programming
Classes versus Prototypes in
Proceedings of the ACM/IEEE
Fall Joint Computer Conference
Dallas, TX, November 1986, pages 36-40
G. L. Dresher
The ObjectLISP USER Manual
Cambridge: LMI Corporation
[Foote & Yoder 1995]
Brian Foote and Joseph Yoder
Architecture, Evolution, and
Second Conference on Pattern
Languages of Programs (PLoP '95)
Monticello, Illinois, September 1995
Pattern Languages of Program Design 2
edited by John Vlissides, James O.
and Norman L. Kerth.
[Gamma et. al 1995]
Eric Gamma, Richard Helm, Ralph
and John Vlissides
Elements of Reusable Object-Oriented
Addison-Wesley, Reading, MA, 1995
[Goldberg & Robson 1983]
Adele Goldberg and David Robson
The Language and its Implementation
Addison-Wesley, Reading, MA, 1983
[Gosling et. al. 1996]
James Gosling, Bill Joy, and Guy
The Javaä Language Specification
Addison-Wesley, Reading, MA, 1996
[Johnson & Foote 1988]
Ralph E. Johnson and Brian Foote
Designing Reusable Classes
Journal of Object-Oriented Programming
Volume 1, Number 2, June/July 1988
[Kiczales, et al. 1991]
Gregor Kiczales, Jim des Rivieres,
and Daniel G. Bobrow
The Art of the Metaobject Protocol
MIT Press, 1991
Artificial Intelligence Laboratory
Vrije Universiteit Brussel
Technical Report 87-2
Concepts and Experiments in
OOPSLA '87 Proceedings
Orlando, FL, October 4-8 1977 pages
[McCarthy et al. 1962]
McCarthy, Paul W. Abrahams,
J. Edwards, Timothy P. Hart,
Michael I. Levin
Lisp 1.5 Programmer's Manual, 2nd Edition
Press, 1965, ISBN 0-262-12011-4
History of Lisp
ACM SIGPLAN History of Programming
Los Angeles, CA June 1-3 1978
[Messick & Beck 1985]
Steven L. Messick and Kent L. Beck
Active Variables in Smalltalk-80
Technical Report CR-85-09
Computer Research Lab, Tektronix,
PCLOS: Stress Testing CLOS
OOPSLA/ECOOP '90 Proceedings
Ottawa, Ontario, Canada
[Roberts & Johnson
Don Roberts and Ralph E. Johnson
Evolve Frameworks into Domain-Specific
PLoP ’96 submission
Brian Cantwell Smith
Reflection and Semantics in a
Procedural Programming Language
Ph. D. Thesis, MIT
Brian Cantwell Smith
Reflection and Semantics in Lisp
Proceedings of the 1984 ACM
Principles of Programming Languages
[Smith & des Rivieres 1984]
Brian Cantwell Smith and Jim des
Interim 3-LISP Reference Manual
Xerox Intelligent Systems Laboratory
Xerox Palo Alto Research Center
Guy L. Steele Jr.
Common Lisp: The Language
Digital Press, 1984
Guy L. Steele Jr.
Common Lisp: The Language
Digital Press, 1990
[Stein et. al. 1988]
Stein, Henry Lieberman,
and David Ungar
Shared View of Sharing: The Treaty of
Concepts, Databases, and
edited by Won Kim and Frederick H.
ACM Press, New York, New York, 1989
The C++ Programming Language
Addison-Wesley, Reading, MA, 1991
[Ungar & Smith 1987]
David Ungar and Randall B. Smith
The Power of Simplicity
OOPSLA '87 Proceedings
Orlando, FL, October 4-8 1977, pages
[Watanabe & Yonezawa 1988]
Takuo Watanabe and Akinori Yonezawa
Reflection in an Object-Oriented
OOPSLA '88 Proceedings
San Diego, CA, September 25-30, 1988
Akinori Yonezawa, editor
MIT Press, Cambridge, MA