Screenshot to React Native: Inside RapidNative's Vision Pipeline

Paste a screenshot of Instagram's feed into a chat box. Seven seconds later, a working React Native screen is rendering on your phone via QR code — the same double-tap-to-like heart, the same story ring gradient, the same tab bar. No Figma import, no manual redraw, no component library shopping.

That's what "screenshot to code" looks like from the outside. From the inside, it's a six-stage pipeline that has to survive a paste event, a page refresh, a compression pass, a monorepo asset path, a model router, and a token-cost budget — before a single line of JSX gets streamed back.

This is a technical walkthrough of that pipeline as it runs inside RapidNative, the AI mobile app builder. If you've ever wondered what actually happens between an uploaded PNG and a rendered <View>, this is that.

Screenshot to app: the goal is that the phone matches the picture, faster than you could have drawn it.

What "screenshot to code AI" actually means

Screenshot to code AI is the workflow where a multimodal large language model — one that can look at an image the way it reads text — interprets a UI screenshot and outputs the source code needed to reproduce it. In RapidNative's case that output is React Native JSX, styled with NativeWind, running inside an Expo project you can preview on a real device.

The trick is not the model reading the image. That part is mostly solved. The trick is everything that has to go right before and after the model call for the output to feel instant, cost less than a coffee, and land in a project you can keep editing.

The pipeline in one paragraph

A user pastes or drags an image into the chat composer. The browser turns it into a File object. Client-side compression shrinks it. IndexedDB holds onto it long enough to survive a page navigation into a freshly created project. An upload endpoint sanitizes the filename, writes it to Supabase Storage, and records it against the project. The generation route sees an image on the last user message and routes to a vision-capable model on OpenRouter — different provider from the text path. A system prompt tells the model to treat the image as the definitive design spec. The response streams back over SSE as JSX, tool calls that write files, and NativeWind classes that get resolved into a real preview. Each of those steps has a failure mode; each of them earned its place because a simpler version broke.

Let's go through them.

Stage 1: capturing the image in the browser

The composer accepts three ways to hand it a screenshot:

• File picker. A hidden <input type="file" accept="image/*"> triggered by a button.
• Drag and drop. The chat textarea listens for drop events with an image MIME type.
• Paste. The most-used one — a paste handler on the textarea reads e.clipboardData.items, walks them looking for the first item whose type.indexOf('image') !== -1, and calls item.getAsFile().

That last one matters more than it looks. Designers and PMs live in a paste-from-Cmd-Shift-4 world. If pasting a screenshot doesn't work, they route around your tool. Here's the shape of the handler that catches it, roughly as it appears in the composer component:

const handlePaste = (e: React.ClipboardEvent) => {
  const items = e.clipboardData?.items;
  for (let i = 0; i < items.length; i++) {
    if (items[i].type.indexOf('image') !== -1) {
      const file = items[i].getAsFile();
      if (file) {
        setImageFile(file);
        e.preventDefault();
        break;
      }
    }
  }
};

Three things worth noting. First, we preventDefault so the image doesn't also try to paste as a broken <img> tag into the textarea. Second, we break at the first image because Mac clipboards sometimes carry the same image in multiple formats. Third, once imageFile is set, the composer renders a preview thumbnail with a remove button — no upload has happened yet.

The composer holds the image locally until you send. Nothing hits the network until you commit.

Stage 2: the pre-project handoff (IndexedDB + compression)

Here's the awkward part. When a new user lands on RapidNative and pastes a screenshot into the marketing-page composer, there is no project yet. The click on "Create App" has to:

1. Create the project row in the database.
2. Navigate to the project editor at a new URL.
3. Still have the image available to upload once the editor mounts.

React state won't survive a page navigation. Session storage struggles with large binary blobs. So the composer writes the file to IndexedDB under a fixed key — pendingImageBlob — before dispatching the project creation. Once the editor mounts, it reads the same key back, uploads the file, and clears it.

// simplified
export async function saveImageToDB(file: File) {
  await set('pendingImageBlob', file);
}

Before saving, the composer compresses. compressImage() reads the file as a data URL, draws it into a canvas at a capped max dimension, and returns a JPEG blob at 0.8 quality. A 4K iPhone screenshot that comes in at 5.2 MB routinely drops to under 400 KB — a 13× reduction that shaves both upload time and downstream token cost, since vision models charge by image tile.

If compression fails for any reason (unsupported format, canvas error), we fall back to the original file rather than blocking the request. Delivering something at all beats optimizing an image that never sends.

Stage 3: storage and the monorepo asset path

Once the editor mounts and the project exists, the image goes to POST /api/upload. Three things happen server-side.

Auth. NextAuth session is required. No project ID, no session, no upload.

Filename sanitization. Names get stripped of anything that isn't a-zA-Z0-9._- (so IMG_2340 (1).PNG becomes IMG_2340__1_.PNG). If the extension is missing, we infer it from the MIME type via a small map — image/png → .png, image/webp → .webp, etc. Then we deduplicate: if the sanitized name already exists in the project, we suffix a nanoid so uploads don't clobber each other.

Storage and aliased paths. The file lands in the projects bucket in Supabase Storage under a structured path — projects/{projectId}/fs/{targetDir}/{fileName}. For projects using the monorepo layout (with a mobile/ workspace), it's placed inside mobile/assets/ so React Native's bundler can resolve it via require(). We also write an alias entry at a flattened path so preview iframes with different working directories still resolve the asset.

Two database writes finish the job: a files row marking it as an external asset (is_external: true, file_type: 'other'), and a project_assets row that tracks the public URL and upload time. The response returns the public URL, the filename, and the file path — everything the client needs to reference the image in the next AI message.

Stage 4: model routing — vision vs text

Now the message goes to the AI generation endpoint. The first thing that route does — before any prompt building, before any tool wiring — is check if the last user message includes an image:

const lastUserMsg = messages.findLast((m) => m.role === 'user');
const hasImage = lastUserMsg?.parts?.some(
  (p: any) => p.type === 'file' && p.mediaType?.startsWith('image/')
) ?? false;

That boolean gates a fork in the pipeline. Text-only requests go to DeepSeek natively — it's cheap, it's fast, and its thinking mode is genuinely useful for reasoning about component structure. But DeepSeek isn't the strongest vision model available, and the vision-capable variants that exist would blow the cost model on every generation.

So image-bearing requests get routed through OpenRouter to a vision-tuned model — the exact model comes from an ai_model_config table with a purpose column, queried via getModelConfigForPurpose('VISION'). That indirection matters: it means we can swap the vision model per environment (or per user cohort) without a code change. When a better model ships, we flip a row.

The routing table looks roughly like this:

Billing follows the same fork. Vision requests are priced separately in the credit calculator because vision tokens cost roughly 5× what text tokens do; hiding that in a single line item would leak money silently.

Stage 5: the vision prompt — image as spec, not suggestion

The prompt matters more than the model. A generic system prompt sent to a vision model produces mush — it treats the image as one more piece of context, weighs it against the text prompt, and averages the two. Averages produce Frankenstein screens.

The system prompt for image-bearing requests contains an explicit instruction, roughly:

When the user attaches an image (screenshot, mockup, wireframe), replicate its visual design as closely as possible — layout, colors, typography, spacing, icons. Use the image as the definitive spec; the text prompt provides supplementary context.

Two words in there are load-bearing. Definitive tells the model that the image outranks the prompt when they conflict — if the screenshot shows a red button and the prompt says "make it blue," the screenshot wins unless the user is explicit. Supplementary tells it not to ignore the prompt entirely; the prompt still supplies intent ("this should be the settings screen"), naming, and behavior that the image can't convey.

A second instruction handles the asset path. If the uploaded image is going to appear inside the app (not just as a design reference), it lives at a real path — mobile/assets/hero.jpg. The prompt tells the model to reference it with require() and specifically not to call the download_asset tool, which would waste a round trip fetching something that's already in the project tree.

The output is real React Native code — editable, exportable, and rendering live via Expo.

Stage 6: prefix caching and the cost of remembering

DeepSeek and OpenRouter both charge less for cached tokens than for fresh ones. Prefix caching means: if the first N tokens of your request are identical to a previous request, the provider charges you a fraction of the normal rate for those tokens.

Two things break prefix caching. Dynamic content in the system prompt (timestamps, user names, anything that changes between requests). And large user-message payloads that shift the position of every subsequent token.

The system prompt is fully static — no interpolation, no dates, no user data. That's a design choice paid for in prompt-engineering flexibility.

The image handling is more subtle. When a user pastes a second screenshot in the same conversation, we don't just append it; we walk backwards through the message history and strip image parts from every user message except the latest one. The reasoning: the second screenshot supersedes the first, but the first is still sitting in the conversation payload, blowing up token count and busting the prefix cache for the entire prompt above it.

// simplified — strips images from all user messages except the last
const stripped = messages.map((msg, idx) => {
  if (msg.role !== 'user') return msg;
  if (idx === lastUserIdx) return msg; // keep latest
  return { ...msg, parts: msg.parts.filter(p => p.type !== 'file') };
});

We do the same with reasoning parts on assistant messages — DeepSeek's thinking traces are useful in the moment but pure ballast on the next turn. Aggressive pruning keeps the cache hit rate above 80% for multi-turn conversations, which materially changes the unit economics.

Streaming: SSE, tool calls, and file writes

Once the request goes out, the response comes back as a Server-Sent Events stream — text deltas, tool calls, tool results. The coding agent doesn't return a monolithic JSX blob; it returns a sequence of tool invocations against a file system. begin_write_file opens a target path, write_file_content streams the body in chunks, db_migration_new applies a Supabase migration if the screen needs backing data.

Each tool result flows back into the SSE stream so the editor UI can render progress: a file appears in the sidebar, a preview iframe reloads, a database schema updates in the inspector. The whole loop from paste to first pixel is typically 3–8 seconds, depending on screen complexity.

Behind that, an Inngest-driven worker persists the SSE stream to Redis so a browser refresh mid-generation doesn't lose the in-flight response. That was one of the earlier lessons: users refresh. A generation that costs $0.20 in tokens should not disappear because someone hit Cmd-R.

Things that broke, and what they taught us

Every stage above earned its place by being the fix for something that didn't work. A few of the more instructive ones:

"The image looked fine in the preview but the model didn't see it." Early on, image parts were being dropped during message serialization when the payload was too large. The fix was compression at the client — and, critically, a size guard on the server that surfaces a clear error rather than silently sending a text-only prompt to a vision route.

"Uploading a PNG named Screen Shot 2024-11-14 at 3.42.18 PM (2).png broke asset resolution." The React Native bundler doesn't love spaces in filenames. Sanitization is not optional.

"Two users' screenshots overwrote each other." Two people uploading screenshot.png in the same project collided in Supabase Storage. The nanoid suffix on dedupe fixed it — with the trade-off that filenames are less human-readable, which the editor UI compensates for with a display-name field.

"The vision model produced great-looking JSX that referenced icons that didn't exist." The prompt now includes an explicit list of the icon libraries available in the template (Lucide, Ionicons), and instructs the model to substitute rather than hallucinate. Real UI parity beats real icon parity.

"A follow-up prompt in the same conversation cost 5× the first." That was the un-pruned image history problem. Stripping old image parts brought it back in line.

Why we bet on multimodal, and what it means for anyone building similar tools

The bet underneath all of this is that visual references are how most humans communicate about UI, and that the fastest path from "I want an app that looks like this" to "here is an app that looks like this" runs through a vision model, not through a chain of Figma imports and code-generators.

That bet has some implications worth internalizing if you're building anything similar. Vision requests are expensive but latency-bounded — the model cost is a rounding error compared to the developer time saved. Pipeline plumbing (compression, caching, routing) is where the actual cost control lives. And prompt engineering for vision is a different skill from prompt engineering for text — the model needs to be told which input outranks which, in words that don't hedge.

For an outside overview of how vision LLMs are being deployed for UI generation, the Vercel AI SDK docs on multimodal messages and Anthropic's vision guide are both good background — they cover the general shape of what tools like this are doing at the model layer, above the pipeline concerns.

Try it

If you want to run a screenshot through this pipeline yourself, open a new project on RapidNative, paste a screenshot into the chat, and hit send. You get 20 free credits, no card required — enough to generate several screens and iterate on them.

If you want to skip the setup and try just the image-to-app path directly, the image-to-app landing page has a direct entry point. Upload a PNG. Preview it on your phone via QR code. Export the React Native code when you're ready to keep building.

That's the end of the pipeline — a rendered screen, back on the phone the screenshot came from. Six stages, one paste.