It seems like there's a strong aversion to using frameworks in the
Clojure community. Other languages might need frameworks, but not
ours! Libraries all the way, baby!

This attitude did not develop without reason. Many of us came to
Clojure after getting burned on magical frameworks like Rails, where
we ended up spending an inordinate amount of time coming up with hacks
for the framework's shortcomings. Another "problem" is that Clojure
tools like Luminus and the top-rate web
dev libraries it bundles provide such a productive experience that
frameworks seem superfluous.

Be that as it may, I'm going to make the case for why the community's
dominant view of frameworks needs revision. Frameworks are useful. To
convince you, I'll start by explaining what a framework is. I have yet
to read a definition of framework that satisfies me, and I think
some of the hate directed at them stems from a lack of clarity about
what exactly they are. Are they just glorified libraries? Do they have
to be magical? Is there some law decreeing that they have to be more
trouble than they're worth? All this and more shall be revealed.

I think the utility of frameworks will become evident by describing
the purpose they serve and how they achieve that purpose. The
description will also clarify what makes a good framework and
explain why some frameworks end up hurting us. My hope is that you'll
find this discussion interesting and satisfying, and that it will give
you a new, useful perspective not just on frameworks but on
programming in general. Even if you still don't want to use a
framework after you finish reading, I hope you'll have a better
understanding of the problems frameworks are meant to solve and that
this will help you design applications better.

Frameworks have second-order benefits, and I'll cover those too. They
make it possible for an ecosystem of reusable components to
exist. They make programming fun. They make it easier for beginners to
make stuff.

Last, I'll cover some ways that I think Clojure is uniquely suited to
creating kick-ass frameworks.

(By the way: I've written this post because I'm building a Clojure
framework! So yeah this is totally my Jedi mind trick to prime you to
use my framework. The framework's not released yet, but I've used it
to build Grateful Place, a community for people who are into
cultivating gratitude, compassion, generosity, and other positive
practices. Just as learning Clojure makes
you a better programmer, learning to approach each day with
compassion, curiosity, kindness, and gratitude will make you a more
joyful person. If you want to brighten your day and mine, please
join!)

What is a Framework?

A framework is a set of libraries that:

Manages the complexity of coordinating the resources needed to
write an application...
by providing abstractions for those resources...
and systems for communicating between those resources...
within an environment...
so that programmers can focus on writing the business logic that's
specific to their product

I'll elaborate on each of these points using examples from
Rails and from the ultimate framework: the
operating system.

You might wonder, how is an OS a framework? When you look at the list
of framework responsibilities, you'll notice that the OS handles all
of them, and it handles them exceedingly well. Briefly: an OS provides
virtual abstractions for hardware resources so that programmers don't
have to focus on the details of, say, pushing bytes onto some
particular disk or managing CPU scheduling. It also provides the
conventions of a hierarchical filesystem with an addressing system
consisting of names separated by forward slashes, and these
conventions provide one way for resources to communicate with each
other (Process A can write to /foo/bar while Process B reads from
it) - if every programmer came up with her own bespoke addressing
system, it would be a disaster. The OS handles this for us so we can
focus on application-specific tasks.

Because operating systems are such successful frameworks we'll look at
a few of their features in some detail so that we can get a better
understanding of what good framework design looks like.

Coordinating Resources

Resources are the "materials" used by programs to do their work, and
can be divided into four categories: storage, computation,
communication, and interfaces. Examples of storage include files,
databases, and caches. Computation examples include processes,
threads, actors, background jobs, and core.async processes. For
communication there are HTTP requests, message queues, and event
buses. Interfaces typically include keyboard and mouse, plus screens
and the systems used to display stuff on them: gui toolkits, browsers
and the DOM, etc.

Specialized resources are built on top of more general-purpose
resources. (Some refer to these specialized resources as services or
components.) We start with hardware and build virtual resources on
top. With storage, the OS starts with disks and memory and creates the
filesystem as a virtual storage resource on top. Databases like
Postgres use the filesystem to create another virtual storage resource
to handle use cases not met by the filesystem. Datomic uses other
databases like Cassandra or DynamoDB as its storage layer. Browsers
create their own virtual environments and introduce new resources like
local storage and cookies.

For computation, the OS introduces processes and threads as virtual
resources representing and organizing program execution. Erlang
creates an environment with a process model that's dramatically
different from the underlying OS's. Same deal with Clojure's
core.async, which introduces the communicating sequential
processes computation model. It's a virtual model defined by Clojure
macros, "compiled" to core clojure, then compiled to JVM bytecode (or
JavaScript!), which then has to be executed by operating system
processes.

Interfaces follow the same pattern: on the visual display side, the OS
paints to monitors, applications paint to their own virtual canvas,
browsers are applications which introduce their own resources (the DOM
and <canvas>), and React introduces a virtual DOM. Emacs is an
operating system on top of the operating system, and it provides
windows and frames.

Resources manage their own entities: in a database, entities could
include tables, rows, triggers, and sequences. Filesystem entities
include directories and files. A GUI manages windows, menu bars, and
other components.

(I realize that this description of resource is not the kind of
airtight, axiomatic, comprehensive description that programmers like.
One shortcoming is that the boundary between resource and application
is pretty thin: Postgres is an application in its own right, but from
the perspective of a Rails app it's a resource. Still, hopefully my
use of resource is clear enough that you nevertheless understand
what the f I'm talking about when I talk about resources.)

Coordinating these resources is inherently complex. Hell, coordinating
anything is complex. I still remember the first time I got smacked in
the face with a baseball in little league thanks to a lack of
coordination. There was also a time period where I, as a child, took
tae kwon do classes and frequently ended up sitting with my back
against the wall with my eyes closed in pain because a) my mom for
some reason refused to buy me an athletic cup and b) I did not possess
the coordination to otherwise protect myself during sparring.

When building a product, you have to decide how to create, validate,
secure, and dispose of resource entities; how to convey entities from
one resource to another; and how to deal with issues like timing (race
conditions) and failure handling that arise whenever resources
interact, all without getting hit in the face. Rails, for instance,
was designed to coordinate browsers, HTTP servers, and databases. It
had to convey user input to a database, and also retrieve and render
database records for display by the user interface, via HTTP requests
and responses.

There is no obvious or objectively correct way to coordinate these
resources. In Rails, HTTP requests would get dispatched to a
Controller, which was responsible for interacting with a database and
making data available to a View, which would render HTML that could be
sent back to the browser.

You don't have to coordinate web app resources using the
Model/View/Controller (MVC) approach Rails uses, but you do have to
coordinate these resources somehow. These decisions involve making
tradeoffs and imposing constraints to achieve a balance of
extensibility (creating a system generic enough for new resources to
participate) and power (allowing the system to fully exploit the
unique features of a specific resource).

This is a very difficult task even for experienced developers, and the
choices you make could have negative repercussions that aren't
apparent until you're heavily invested in them. With Rails, for
instance, ActiveRecord (AR) provided a good generic abstraction for
databases, but early on it was very easy to produce extremely
inefficient SQL, and sometimes very difficult to produce efficient
SQL. You'd often have to hand-write SQL, eliminating some of the
benefits of using AR in the first place.

For complete beginners, the task of making these tradeoffs is
impossible because doing so requires experience. Beginners won't even
know that it's necessary to make these decisions. At the same time,
more experienced developers would prefer to spend their time and
energy solving more important problems.

Frameworks make these decisions for us, allowing us to focus on
business logic, and they do so by introducing communication systems
and abstractions.

Resource Abstractions

Our software interacts with resources via their abstractions. I
think of abstractions as:

the data structures used to represent a resource
the set of messages that a resource responds to
the mechanisms the resource uses to call your application's code

(Abstraction might be a terrible word to use here. Every developer
over three years old has their own definition, and if mine doesn't
correspond to yours just cut me a little slack and run with it :)

Rails exposes a database resource that your application code interacts
with via the ActiveRecord abstraction. Tables correspond to classes,
and rows to objects of that class. This a choice with tradeoffs - rows
could have been represented as Ruby hashes (a primitive akin to a JSON
object), which might have made them more portable while making it more
difficult to concisely express database operations like save and
destroy. The abstraction also responds to find, create,
update, and destroy. It calls your application's code via
lifecycle callback methods like before_validation. Frameworks add
value by identifying these lifecycles and providing interfaces for
them when they're absent from the underlying resource.

You already know this, but it bears saying: abstractions let us code
at a higher level. Framework abstractions handle the concerns that are
specific to resource management, letting us focus on building
products. Designed well, they enable loose coupling.

Nothing exemplifies this better than the massively successful file
abstraction that the UNIX framework introduced. We're going to look at
in detail because it embodies design wisdom that can help us
understand what makes a good framework.

The core file functions are open, read, write, and
close. Files are represented as sequential streams of bytes, which
is just as much a choice as ActiveRecord's choice to use Ruby
objects. Within processes, open files are represented as file
descriptors, which are usually a small integer. The open function
takes a path and returns a file descriptor, and read, write, and
close take a file descriptor as an argument to do their work.

Now here's the amazing magical kicker: file doesn't have to mean
file on disk. Just as Rails implements the ActiveRecord abstraction
for MySQL and Postgres, the OS implements the file abstraction for
pipes, terminals, and other resources, meaning that your programs
can write to them using the same system calls as you'd use to write
files to disk - indeed, from your program's standpoint, all it knows
is that it's writing to a file; it doesn't know that the "file" that a
file descriptor refers to might actually be a pipe.

Exercise for the reader: write a couple paragraphs explaining
precisely the design choices that enable this degree of loose
coupling. How can these choices help us in evaluating and designing
frameworks?

This design is a huge part of UNIX's famed simplicity. It's what lets
us run this in a shell:

# list files in the current directory and perform a word count on the output
ls | wc

The shell interprets this by launching an ls process. Normally, when
a process is launched it creates three file descriptors (which,
remember, represent open files): 0 for STDIN, 1 for STDOUT,
and 2 for STDERR, and the shell sets each file descriptor to refer
to your terminal (terminals can be files!! what!?!?). Your shell sees
the pipe, |, and sets ls's STDOUT to the pipe's STDIN, and the
pipe's STDOUT to wc's STDIN. The pipe links processes' file
descriptors, while the processes get to read and write "files" without
having to know what's actually on the other end. No joke, every time I
think of this I get a little excited tingle at the base of my
spine because I am a:

This is why file I/O is referred to as the universal I/O model. I'll
have more to say about this in the next section, but I share it here
to illustrate how much more powerful your programming environment can
be if you find the right abstractions. The file I/O model still
dominates decades after its introduction, making our lives easier
without our even having to understand how it actually works.

The canonical first exercise any beginner programmer performs is to
write a program that prints out, Wassup, homies?. This program makes
use of the file model, but the beginner doesn't have to even know that
such a thing exists. This is what a good framework does. A
well-designed framework lets you easily get started building simple
applications, without preventing you building more complicated and
useful ones as you learn more.

One final point about abstractions: they provide mechanisms for
calling your application's code. We saw this a bit earlier with
ActiveRecord's lifecycle methods. Frameworks will usually provide the
overall structure for how an application should interact with its
environment, defining sets of events that you write custom handlers
for. With ActiveRecord lifecycles, the structure of before_create,
create, after_create is predetermined, but you can define what
happens at each step. This pattern is called inversion of control,
and many developers consider it a key feature of frameworks.

With *nix operating systems, you could say that in C programs the
main function is a kind of onStart callback. The OS calls main,
and main tells the OS what instructions should be run. However, the
OS controls when instructions are actually executed because the OS is
in charge of scheduling. It's a kind of inversion of control, right?