Teaching an agent to think in 3D space.

Why scripting was tempting

Every frontier model has been trained on years of Blender Python (bpy) examples scraped off the open web. So when you ask one for “a low-poly tree blowing in the wind,” it produces something plausible on the first try. No tool layer to build, no spatial reasoning to teach — just generated code, piped into Blender, and you get a render. This is the whole pitch of projects like Blender-MCP. It works. It works enough to be misleading.

The wall — bpy in Blender 5.0

I was on Blender 5.0 because it’s the newest, has the features I want, and is what the project will ship against. The model wasn’t. Frontier training data lags releases by a year or more, so every script was written against an API that had already moved underneath it. The error log filled up with the same shapes of failure, over and over.

The cost trap

The two obvious fixes both lose:

1. Use the very latest model. Slightly better API knowledge, but you’re paying flagship-tier rates for something as cheap as a sword. And the model still lags Blender by months.
2. Stuff the prompt with current API docs. Works once or twice, but you’re paying for tens of thousands of input tokens on every single call — for a problem the docs don’t actually solve. The model still hallucinates around them.

Either way, you’re paying more, every call, forever, just to keep up with whichever Blender version you’re on. That math doesn’t work for a product.

The pivot — tools the developer maintains

The shift was: take an older, cheaper Gemini and stop asking it to write bpy. Have it call tools — small, well-specified functions I maintain — that wrap whatever the current Blender API is.

The win that made this real for me: I could write tools the model couldn’t have generated on its own — scatter_on_surface, array_along_curve, shrinkwrap_modifier — with the edge cases handled in code, once, by a developer. The model just decides when to call them.

The trade-off, honestly

Tools are cheaper to run and far more reliable, but they’re harder to build. With scripting, the model handles the spatial reasoning implicitly — it just emits coordinates that look right. With tools, you have to teach the model spatial reasoning, because it’s now choosing tool arguments instead of writing free-form geometry code. That’s the entire reason the next two sections exist:

• Selection — how does the model point at which geometry?
• Transformation — how does it reason about orientation and scale?

Both problems were free in the scripting world. Both had to be re-earned in the tools world. The payoff is everything downstream — cost, reliability, maintainability — works.

What I’d do differently — the hybrid future

Scripting isn’t dead. The next iteration is a hybrid: script-first for the tasks where models are genuinely strong (alignment math, scale ratios, parametric placement — coding, basically), with tools as a fallback when the script errors out instead of just retrying. Models reason better about code than about 3D space, so use them where they’re sharp; use deterministic tools where they’re not.

I’ve always been inspired by Gundam model kits. I’ve built a few by hand, and I wanted the agent to eventually get there — which meant it needed a way to grab specific geometry, not just whole objects. A connector pin. The inner lip of a pot. The eye slits on a helmet visor. None of those work without selection.

Step 0 — the upstream dependency: see everything

Before I could touch the selection logic at all, I had to fix something further upstream. The early viewport setup gave the agent four orthographic angles — front, back, left, right. That’s enough for a cube. It’s nowhere near enough for a Gundam, where every interesting feature is hidden from at least one of those four views.

The cameras are pre-defined and parented to a single rig empty, so framing an object means moving one transform — not re-positioning eight cameras independently. Distance and ortho-scale are derived from the target’s bounding box, which gives raycasting a known, normalized projection to work against.

The first approach — a query engine

With the angles fixed, my first attempt at selection was to let the agent describe what it wanted. Named presets — TOP, BOTTOM, FACING_UP, BOUNDARY — plus a small DSL on top: pick a property like centroid.z or normal.x, an operator (GT, LT), a threshold normalized to the object’s bounds. The engine resolved the query against a mesh snapshot and returned matching geometry.

// "Top faces of a donut, facing up"
select_by_query({
  object_name: "donut",
  element: "FACE",
  logic: "AND",
  conditions: [
    { property: "centroid.z", op: "GT", value: 0.7 },
    { property: "normal.z",   op: "GT", value: 0.5 },
  ],
});

For simple objects this was beautiful. The agent never touched a vertex index — it described intent, the engine deterministically resolved it. But the moment the geometry got complex, the cracks showed up everywhere.

Where it broke down

The donut problem. “Select the top half” is fine. “Select the inner ring” needs another preset. “Select the rim, not the floor” needs a third. Every new selection pattern needed a new preset or a more elaborate condition combination — TOP_RIM, BOTTOM_CENTER, OUTER_SIDES, INNER_RING — and the disambiguation logic got worse with each addition. The preset library kept growing.

The Gundam problem. A query engine works when the geometry has clean, separable regions. A Gundam doesn’t. To pick out just the V-fin, or just the left eye slit, or just the inner edge of the visor, you’d need a preset for every named part — V_FIN, FACEPLATE, LEFT_EYE, RIGHT_EYE, VISOR_INNER — and even then the agent would need to know each part’s exact name. That’s a hand-curated dictionary forever, and it doesn’t generalize beyond Gundams.

The replacement — visual click-select

The shift was conceptual. Stop having the agent describe geometry in words. Give it the rendered image — the eight angles it already has — and let it point.

The agent advantages compounded immediately. No graph in the prompt — we don’t feed it “here are the 240 named parts of this Gundam.” Token cost drops — an image is fixed-size, a part graph isn’t. The right modality — modern models reason far better about where something is in an image than about a JSON tree of mesh parts.

The hard part underneath the elegance

Pointing sounds simple. Making it land on the exact pixel the agent meant is not. Three pieces had to slot together:

1. Fixed framing. Each viewport tile has a known ortho-scale derived from the target’s bounding box. The same world point always lands on the same pixel — no projection drift between captures.
2. Per-tile UV coordinates. The agent emits [u, v] in 0–1 within a single tile rather than absolute pixel coordinates of the stitched image. Smaller numerical range, fewer sources of error.
3. Calibrated raycast. Per-view axis flips and a Y-flip (image-Y vs OpenGL-Y) are applied before firing the ray. When the ray misses, screen-space fuzzy matching falls back to the nearest vert / edge / face within ~50px — the same algorithm Blender’s native selection uses.

// Agent emits a coordinate inside a named viewport tile.
// [0.5, 0.5] = center, [0,0] = top-left, [1,1] = bottom-right.
click_select({
  camera: "iso_front",
  click:  [0.62, 0.34],   // per-tile UV
  element: "VERT",
  object_name: "gundam_head",
});

I borrowed heavily from the browser-automation world, which has been solving “click on a thing in a rendered image” for years. Game-engine raycasting handled the hit; the calibration plumbing — resolving UV to a stable world ray — was the original work.

The first transform tool — a Swiss Army knife

I’d modeled the first transform API on Blender’s own N-panel “Item” section, where Location, Rotation, and Dimensions all live under one heading. The first iteration mirrored that — one transform tool with an operation enum.

// Modeled on Blender's Item panel: Location / Rotation / Dimensions.
transform({ operation: 'TRANSLATE', name: 'Cube', value: [1, 0, 0] })
transform({ operation: 'ROTATE',    name: 'Cube', value: [0, 0, 90] })
transform({ operation: 'SCALE',     name: 'Cube', value: [2, 2, 2] })

This became the test bed for the harder question: how do you get a language model to reason about where things go in three dimensions?

Bug #1 — orientation (the USB test)

I asked the agent to model a USB connector. It understood the components — body, shell, four pins — and it correctly built the body along the X axis. But the pins came out vertical along the Y axis, lined up in a row instead of arrayed across the connector face along X.

I tried three ways out before settling on what actually worked.

After three iterations chasing a way to avoid raw axis numbers, I went back to numerical XYZ. The fix wasn’t the vocabulary — it was the process around it.

Bug #2 — scale vs. dimensions

For weeks, scenes had broken proportions: table legs that were too long, mountains and the rocks sitting on them at identical scale. I assumed it was a model-intelligence problem. It wasn’t. It was a tool-design problem.

I added a new tool, resize, that operates on world-space dimensions in meters, and split the unified transform back into focused, single-purpose verbs.

move(name, value, mode)        // translate by [x,y,z]
rotate(name, value, mode)      // rotate by [x,y,z] degrees
scale(name, value)             // multiplier — "make it 2x bigger"
resize(name, dimensions)       // absolute meters — "make it 20m wide"