writing memos

I was planning to write a memo about AI for design for a couple of days now, but I found myself unwilling to start. I think this was because I just didn’t have that many undistilled thoughts. As a team, we’ve discussed this thread/overarching motivation many times, and although it would be nice to put it into writing, I didn’t really feel like there was much to gain from condensing all our thoughts into words. Sure, it would be nice from a memos sake, but I wasn’t confident that I’d realize new insights for our team through the process of writing. Without confidence in the reward/learnings from this task, I subconsciously sought other, more impactful tasks. That being said, I’m finally summarizing my thoughts on AI for design, although it is not going to be as comprehensive as I originally intended.

augmenting knowledge workers

During our deep dive into generative AI at San Diego, we eventually settled on exploring applications of LLMs to augment knowledge workers. An example of this is Github Copilot for software engineers—rather than completing replacing a software engineer, Copilot seamlessly integrates into VSCode to provide you autocomplete suggestions while you’re coding. It’s super useful as each suggestion can save you 25 minutes of googling stackoverflow threads/reading documentation to write one-off, logically simple code. It’s also especially great for filling out parameters in function calls—typically, you’d have to go read some API docs, understand what parameters are required, and go through and type everything out. Copilot removes all that busywork and provides code that’s as clean/elegant as that of a senior SWE—it truly feels like a magical experience. This is evident from Copilot’s over a million users—the more mind boggling stat is that 40% of their users’ code is generated from Copilot’s suggestions.

There are several key points about Copilot that stand out to me. First off, AI generated code is hardly a new idea—having an AI programmer, if possible, is clearly invaluable. The issue is that AI is nowhere close to being able to consistently generate flawless code from scratch. The obvious UI/UX of asking a bot to write code and then said bot returning you code to copy and paste doesn’t feel magical because the result will be garbage a significant portion of time. The key insight that the Copilot creators realized was to only deliver suggestions when the model is supremely confident that it’s correct—from the programmer’s perspective, most of the time you don’t even realize that Copilot is there. But once in a while, it’ll deliver you a suggestion that’ll save you half an hour of work—that sort of experience is magical and addicting.

Along these lines, the second insight is that it’s quite hard to replace entire humans for generative work, at least with our current AI technology. A lot of focus is placed on the idea of replacing human workers—the lowest hanging fruit would be low-value workers (people who do low-skill, repetitive tasks) rather than knowledge workers (i.e. software engineers). This is not only hard, but might also be inefficient from a business perspective. Replacing 100% of a worker might not deliver as much value as making a knowledge worker 40% more efficient (like in the case of Github Copilot). The latter is also just much easier to do because you have a human in the loop to capture the mistakes/context that the AI simply doesn’t have—as you scale, you can continue to improve your model with human feedback. This sort of human in the loop deep learning seems to be the most promising application for AI embedded products—my team and I are paying very close attention to reinforcement learning with human feedback (RLHF), and I’d urge others working in this space to do the same.

Finally, the reason why this generative suggestion model works so well is that writing code is often an NP task (here, the analogy is to the P=NP problem/computational complexity). It might take a senior software engineer to understand the codebase intimately and write high quality code, but even relatively junior developers can recognize good/functional code. This sort of task, where it’s easy for many to verify that a generated solution is good but difficult to come up with the solution in the first place, is an application where AI can provide a ton of clear value. For senior software engineers, you remove the tedious aspects of programming. Whereas for junior software engineers, you suddenly unlock the ability for them to produce code at an expert level. There are many examples of this type of “NP” task: math proofs (hard to come up with, easy to show it’s correct), art, writing, etc. Design falls into this category which is why we were interested in it.

The four of us are varied in our technical backgrounds, but none of our backgrounds are in design. That being said, we still use Figma a lot for design stuff, and one of our favorite past-times in Oregon was to critique DevOps startups’ landing page designs. It’s easy for us to recognize good design, but extremely difficult for us to utilize the necessary tools to create good design. AI is perfect for augmenting this sort of process.

Designers are also similar to software engineers in that they are knowledge workers, and in tech at least, provide a lot of economic value to companies. Being able to augment these sorts of workers, from beginners to expert UX designers, would likely create a lot of corresponding value—you can’t get people to pay for your product unless you create a lot of value for them. Or maybe you can, but that seems to involve a lot of conning that I’m unfamiliar with.

We also want to capitalize on the “magic” associated with generative AI—having a product that’s simultaneously a magical experience does wonders for sales/viral growth. Yes we could have an AI product that has solves a super salient problem for someone, but I think a design tool that just spits out good designs with the power of generative AI is just undeniably super cool. From a technical perspective, it definitely stretches the limits of what’s possible—this makes it really interesting/intellectually stimulating to work on, and we’re all excited to see where this takes us.

next steps

We are all painfully aware by this point that the odds that we pivot from any given idea is high. That being said, if we wish to learn more about the space/potentially develop infrastructure for AI-embedded products, the process of building something like this out will be invaluable.

Along the lines of this product, I would like to validate some hypotheses surrounding this product, namely whether or not an AI-embedded design tool actually solves any problems for people. There’s a lot of technical exploration we have to do as well, both on the data/AI side, but also on the more traditional software engineering side—I’m actually highkey unsure whether or not this is even technologically possible to build this year. However, there’s still a lot of work to be done before we dig into the weeds of building such a product out—we want our learnings to be as efficient/direct as possible, and I doubt untangling the intricacies of the Figma API is the most effective use of our time if we do end up pivoting yet again.

Anywho, read my startup logs if you want a better sense of what we work on on a day-to-day basis! I can’t make any promises as to how interesting it is—there are indeed a lot of photos of us just sitting around the house.