I spoke with Mary Rose Cook, the researcher-in-residence at Ink & Switch. We talked about her recent work on prototyping, and her thoughts on prototyping tools/workflows in general.

You can find Mary on her Twitter or her website. Her recent research notes can be found on her Notion page.

Enjoy!

Editor's note: This transcript has been lightly edited for clarity. All emphasis (italics, bold, underlines) were added post-conversation.

Alex
Hi, Mary - thanks for agreeing to this conversation.
In your previous work, you have Isla and Code Lauren, which were for educational purposes - specifically for kids.
And now you're working on prototyping tools for building software.
Both seem like 0-to-1 type work, but I was struggling to see the common thread between the two. What would you say is the common thread running through your work?

Editor's note: Here are the links to Isla and CodeLauren.

Mary
I think the thread is not being blocked in getting to build what you're trying to build.
Some people are blocked because they can't program or they're not very sophisticated programmers yet.
Sometimes people are blocked just by how long something takes to do. And that blocking has quite a tricky effect on all the ideas that you never even try.
If you could just prototype extremely quickly, you'll be much more likely to try it.
It's not necessarily that that idea is the killer idea that you really need to get to. It's that that is probably going to be a jumping off point for generating the really, really good idea that comes on later.
Alex
Something that I think programming is really good at is teaching a kind of "iterative" approach to problem solving.
Prototyping - to me - seems like it almost demands this iterative-type thinking. Because otherwise, you'll get blocked.
Is it possible to "force" this iterative type of thinking upon the user via a good medium?
Mary
You can often prototype ideas very simply just on paper, just as a sketch or something.
That mode of sketching something, and looking at it, and iterating on it - I think that's a pretty natural mode for designers. I've seen them be very comfortable with that.
Whereas - this is a massive overgeneralization - programmers are more about incremental building towards a specific thing.
A programmer will often know where they're going, and there are going to be a lot of steps to get there. So they'll hack in a bunch of things to get the final thing basically working.
But that's not really iterative exactly. In the sense that programmers are not iterating on the final result.
They're more just building in smaller steps towards the final result.
A designer might do 8 Figma sketches really quickly, building a big mess. And then they'll just keep on duplicating, tweaking.
Whereas code - it's extremely hard to do that with.
So there is definitely a medium problem there. But it is also a mindset thing, a personal skill set thing.
Alex
I'm curious -
I'm not a big gamer or anything.
But I know you derive a lot of inspiration from games and game design.
I would think that game designers: they're a good marriage of design-type work and programming-type work.
I'm wondering how they might operate in this intersection.
Mary
I've seen - really weirdly - both poles end in success.
Which can be a little strange.
For example, there's this one company - Naughty Dog - which made The Last of Us. When they built that game, it wasn't fun until the last third of development.
And they spent years on it.
So they were putting all of this time, effort, and money into building this game - and they still didn't know if it was going to be fun.
They just had a kind of faith that they would find a way to make the ingredients fun.
So I've seen that model - where I'm sure they were doing a lot of prototyping to get to the fun - but they're still building this very big complicated thing with a lot of investment upfront.
There's another company called That Game Company, which made a game called Journey.
They start with what's called "gray box". It's essentially just gray squares on a screen and some interactivity. And it ends up as an incredibly rich sensory experience. But it all just starts off as a 2D gray box moving on the screen!
And that's all about iterating on the core gameplay.
These two approaches appear - to me - to be very different.
All of this is a long-winded way of saying that game designers prototype at varying degrees of fidelity.
Some are building in the manner of The Last of Us, where they prototype on the side, and when they find something good, they'll bring it into the main experience and hash it out.
Whereas with others, they just don't do anything until they've got something that's fun - even as gray squares.
Alex
That's a good segue, actually.
I want to talk a little bit about like expressivity. I'm not sure your opinion on this, but for a prototyping tool, I would think that you would want very expressive primitives.
A programming language is extremely expressive, but Excel - obviously its primitives are not as expressive, right?
But because programming is so expressive, it's much more complex and hard to get a grasp of.
Whereas with Excel, it's much simpler to grasp because of its less expressive primitives.
With prototyping tools, how do you balance this tension between expressivity and complexity?
Mary
I don't know if you've read A Small Matter of Programming.
Alex
That's how I thought of the question, yea!
Mary
Yeah, so the way she puts it - in my memory - is that the end-user programmer ideally programs in their own domain.
An accountant works with deposits and withdrawals and whatever. Those will be the most low-level units that they have to work with.
In terms of the expressivity of a prototyping tool, a specific tool will be expressive for a person if the primitives are in their domain.
For example, a web prototyping tool is probably going to have inputs and some sort of way representing trees as a layout. Whereas if somebody were building a game - that would be way less important and there would be other primitives.
It's mainly about matching the prototyper to the primitives.
A good example of a prototyping tool that does this well is Axure. It makes bringing in inputs, laying things out, and creating lists of things very easy.
Whereas Figma, it's not nearly as good at that sort of thing.
You as a programmer can probably do a better job of coming up with primitives that will apply to more domains or be able to be composed into more domains.
But my gut says that the tools that are successful start off with a specific thing they're trying to enable. If they are successful, then with some luck it will be adaptable to other environments.
Alex
Feels like iteration all over again.
Mary
Yeah - a good example is Excel, actually.
It was supposed to be a spreadsheet, right?
And then accidentally, it's quite a good to-do list maker and a data storage mechanism.
But that was a total accident.
The original paper spreadsheets were just drawn that way because they needed grids of numbers. It was just pure chance that it happened to be that that works well for making lists of things and data and stuff like that.
I do think that there's a lot more luck in something being adaptable to a different domain than it seems at first glance.
Alex
That reminds me of something you wrote on your Notion two or three years ago.
I forget the exact wording of the question - but you asked if there were any examples where someone had created their own tools and those tools became successful or something like that.
Given that that was asked three years ago - have you found more answers to that question? Any iterations on the question itself?

Editor's note: The post in question is called Have people produced great work using highly expressive tools they invented?

Mary
I actually haven't found that many examples. I guess programmers are really the best examples because they build their own scripts.
Musicians building their own instruments - that seems to happen a little bit.
Ableton Live is also a good example, which I think I mentioned in the post.
It's surprising uncommon to find anything really systematic - it's always seems to be more like "this is a little thing that I use in this specific case".
But it's not a structure in which I map in my entire workflow.
Alex
I guess there needs to be a really good marriage of someone who is a tool builder and is also a tool user.
Mary
Yeah.
But I think also - probably what I didn't see when I wrote that at the time - is that the mode of starting off building something half for yourself, half for other people -
that's by far the most common for "success".
Ableton Live started out as something for the people who created it. But it very rapidly became a tool for other people to use and that influence really guides future development.
Where the line is between it being a personal tool and it turning into something for someone else. But it seems like it's quite common for tools to walk along that progression from personal to "commercial".
Alex
"Personal" becoming a little more "general" -
That sounds very similar to abstractions.
What is the limit to how much you can abstract things away? Because at a certain point - the abstraction almost becomes more "probabilistic" rather than "deterministic".
Mary
Do you mean that at some point it becomes so abstract that it loses meaning?
Alex
A bias is a good example, I think. Like confirmation bias or hindsight bias.
It's not necessarily that any one person might have it - but it's like a statistical "abstraction" because it's a generalization in the "average" case.
Whereas something like a while loop or recursion - those are patterns that are very set in stone.
Mary
The point being that there's only one way to interpret what a while loop is - but there's more interpretability in the more abstract things?
Alex
Yeah, that's a good way of putting it. The further you abstract things out, the harder it gets to actually reason about what the abstraction is.
Mary
I do think that's right - I worry about this in my own work myself.
I don't know if you've read Donald Norman's stuff, but he has this concept of knowledge in the tool versus knowledge in the head.
For example, take a lightswitch.
A lightswitch knows how to flip on and off electricity to the light that you're trying to flip on. But there's a certain amount of knowledge that the user needs to know that, first of all, that's what it's for. And second of all, you press down to turn it off or whatever.
The tool can still support with applying that knowledge. For example, a lightswitch only goes up and down.
Still, the user needs to know "Ah, I flip it down to reverse the status of the light up there."
When I see really expert designers who often use sketches, they will do a sketch that doesn't have much information in it. From an abstract perspective, it's just a few boxes with a squiggle or something.
But they will have a very rich picture in their head of what that means. And they'll also be able to simulate what that experience will be like when mapped to a program.
Whereas me, as an unskilled designer - I'll fail to do neither. I'll fail to figure out exactly what the constraints are that are expressed by this diagram, and also to simulate the eventual effects in the real software if it were built.
I think a big part of it is building up that skill - essentially always executing prototypes in your head.
You've got all this richness in your head that makes the prototype much more expressive than it would be if it were run in my head.
I think there's a very strong relationship there.
Yes, abstract can be worse because it's prone to misinterpretation or it just doesn't necessarily convey all the information that it's supposed to convey.
But actually abstraction is fine, as long as the knowledge required to flesh out the abstraction is in the viewer's head. It really depends who the audience is.
A designer can just sketch to themselves. But when they present it to someone else, they're probably going to want to flesh it out in order to give that full experience.
The question you've raised - it's possibly one of the core tensions of prototyping: striking that balance between quickness of expression and communication of the eventual experience.

Editor's note: Donald Norman is the author of The Design of Everyday Things.

Alex
I suppose all of this takes on a different dimension when you're co-designing. Have you thought about this with "multiplayer prototyping"?
Mary
Yeah, I have thought about it.
In the ideal case, the prototyping medium will express and constrain some of those degrees of freedom that a more abstract medium might not.
So, you can draw anything you want on paper, whereas if you try to draw a box in code, you're probably just going to get an actual box that's going to have certain traits.
Ideally, the prototyping medium would support the constrained degrees of freedom that you have to be constrained by in the final medium, whilst at the same time, letting you be very expressive.
Again, you touched on what feels like another central tension: limiting expression for the purpose of making something more concrete and specific, versus overly constraining degrees of freedom to the point of crushing ideas and crushing expressivity.
There's a very good book on this called Sketching User Experiences. It's about different types of sketch and it goes a lot into this - these tensions between fidelity and communication. It's excellent.
Alex
Thanks, I'll check it out.
How much of a prototyping tool do you think should be about putting the simulator in your head into actual real life versus allowing you to explore the space of your ideas, so to speak?
Is it counterproductive for a prototyping tool to allow for latter?
I can easily imagine a prototyping tool where you put in a simulation and then some AI tool will inject like some randomness so that it can "spark ideas".
Mary
There's two modes, at least.
One mode is validation: does this solve the problem I'm trying to solve?
Another mode is generation, which is more creative help.
I think prototyping tools are almost always in the former mode. The experiments I've seen in the latter mode, around generative prototyping, are much more just experiments.
I suspect that you probably are in one mode or the other quite a lot of the time. And I think it would be very interesting to explore. It would be extremely useful in a prototyping tool and an ideal one.
To take a sort of silly example - if I draw a square and then I get around to start thinking about what color is the square, then the generative tool would confront me with a decision or suggest a decision.
Whereas the "problem-focused" one would not. It would just take the color as read.
I do think there are some tools that do this quite well, where they can quietly suggest whilst not getting in the way.
One good example of this is Copilot.
It fills in some gray text of what the code could or should be for you, and it shifts the rest of the code down a bit to make space. But beyond that, it doesn't really change too much. It's very quiet.
It doesn't get in the way too much if that's not what you want, and you can just carry on typing and get rid of its code if it suggests the wrong thing.
So to me, Copilot is quite a good example of it's both generative and problem-solving in the same interface. I think it's very clever. And so it definitely seems possible.
In the end, my dream prototyping tool is just something where, as quickly as possible, I can just put what is in my head on the screen and try it out. And if that were extremely quick, then generation sounds great, but it would be a secondary thing.
Right now, you either do the quick way, in which case you make mistakes later because you don't think about things - or you do the slow way and you get massively over-invested in the boat you just made.
Alex
Sounds familiar.
What do you think a really bad prototyping tool would be like? The worst one you can think of - what is a couple features that you can imagine would just make the experience horrible and make you want to leave?
Let's say for designing web apps, to make it constrained.
Mary
One where you can't do programmatic generation of HTML.
Most tools you can say "oh, I want a title here", and "oh, I want some body text here", and so on.
But you can't actually work with data in any kind of programmatic way.
You can't say, for example, make this a list where I can add new items to the list, and it will grow in the expected way.
What's crazy is that for almost every web program, that's how it works. It's insane.
Even Axure, which is my favorite, it has a go with lists.
It tries a bit, but with nested data structures - forget it. Even the list handling is very rigid. It's like a product that's been designed in an essentially iterative way -
"Oh, we should add a thing to do this and another to do that"
- but without a holistic understanding, or a holistic rethink of the way it should work.
It's surprising to me how many prototyping tools that run in the computer are not programmatic.
But at the same time, there's obviously a real trap there on the other hand - where they start from the programming, which is basically just like having to do the real code. When you start with the programming, working with the data is a complete nightmare. And it's very hard to get anything on screen. You're very rigid and trapped, mostly.
So the other extreme of writing code, which will always get you to what you want to express, has a lot of other problems, too.
Alex
Do you have an experience of trying to do something in Axure, and you were just like "okay, I hate this"? Is there a defining moment?
Mary
I do.
At Airtable, you can write little programs by filling out forms.
So say every time a new record is created in the database, I want to send an email.
You write that program by selecting the when record created action, and then you select the send an email action, and then you just fill in some details like the email address, which table you care about, and stuff like that.
It's sort of forms-based programming, and it worked pretty well.
When I was working there, we were trying to add control flow to this. It was me as a programmer, another programmer, and then the designer.
We were trying to figure out - what are the gotchas of this UI that we have when trying to layer in even a single get if-else statement, right? There was some scoping stuff we had to think about, the clauses, and so on.
I tried to process all this in Axure - and essentially I was having to simulate all of those gotchas in my head.
I was having to create a screen where they haven't yet selected a conditional, and then create another screen where they have selected it, and then create another screen where they've selected the else, and so on.
Now, it was helpful, because it still forced me to think it through.
So it was better than just starting to work on programming it, but it didn't really get us much.
In the end, I just ended up building it as a React app, where I simulated the real logic of the eventual interface we were thinking about, and then, boom, all the problems were exposed.
Now, the Axure version probably took a couple of hours, and the React version took more like a day. But that investment was worth it.
With just Axure, we would have ended up being like -
"oh, we found a few problems"
- and then we would have implemented it and found all of the rest of the problems when we actually built it.
So something where essentially the algorithm contains hidden information about how it's going to work - that is quite hard to figure out in advance. Because you just need the medium to be computational.
Which Axure isn't really.
It's still my favorite, but for anything algorithmic, it's really not that great.
Alex
Wow, that sounds like a pretty frustrating experience.
Well, I think we're just out of time.
Again, thank you for agreeing to speak with me, it's been fun!
Mary
Thank you, Alex. It's been great.