Jenga Mosaic — Kinova Gen3 pick & place

What the system does

The robot is shown a partially built 2D mosaic of Jenga planks. A camera captures the workspace, a vision module detects each brick's position and orientation, a pattern module identifies the current tessellation and predicts where new bricks should go, and then the arm picks bricks from a reserve pile and places them — one at a time — until the pattern is fully expanded.

My two modules sat at the bottom of this pipeline: once the system knew where to put the next brick, my code had to find a suitable brick to pick up and then physically execute the move.

My role

The team split the work into six tasks across four people. I took tasks 4 and 5.

Task 4 — new brick picking. Given a camera image of the reserve pile (a loosely scattered heap of Jenga planks), identify each brick's position and orientation so the arm can approach it correctly. The output is a (position, orientation) tuple the arm can use directly. All reserve bricks are laid flat with the same face up, which simplifies the grasp to a single in-plane rotation angle.

Task 5 — robot arm movement. Given a source (position, orientation) and a destination (position, orientation), command the Kinova Gen3 to move the brick from one to the other without hitting the existing structure. This covers both normal placement moves and correction moves: if a brick landed in the wrong position or at the wrong angle, the arm picks it up from its actual pose and re-places it at the target pose.

Because I was working off-site from the Robohub where the physical robot lives, I passed my code directly to the group member running on the arm — which is why my GitHub account doesn't appear in the commit history.

Arm movement in detail

The Kinova Gen3 is controlled through the Kortex Python SDK. Each move is a Cartesian action: a target (x, y, z) and a target orientation (roll, pitch, yaw) sent to the base controller. The arm has hard joint limits and a limited Cartesian workspace, so a direct single-step move from A to B often fails or finds no IK solution.

The fix is to decompose each move into l equal sub-steps along the straight line in Cartesian space, sending incremental deltas (x/l, y/l, z/l) and (ox/l, oy/l, oz/l) per step:

for i in range(l):
    success &= example_cartesian_action_movement(
        base, base_cyclic,
        x/l, y/l, z/l,
        ox/l, oy/l, oz/l
    )

The step count l is chosen to minimise the maximum single-joint displacement per step — keeping motion smooth and within the arm's velocity limits. A camera-to-robot-base coordinate transform is applied to convert pixel coordinates from the vision module into the arm's reference frame before any move is commanded.

Demo

Three clips of the system running in the Robohub:

What was tricky

Working entirely off the physical robot was the main constraint. I had to write and reason about movement code without being able to run it — all testing happened when my teammate loaded it onto the arm in the Robohub. That meant being precise about units, coordinate frames, and edge cases up front rather than iterating at the machine.

The step-count decomposition also needed tuning: too few steps and the arm hits a joint limit mid-move; too many and execution time grows to the point where the demo becomes impractical. The right value was found empirically by my teammate running the arm.

The correction move case — where the arm must recover a misplaced brick — requires knowing the brick's actual pose, not the intended one. That feeds back from the vision and feedback modules (Victor's work), so the interface between those modules and mine needed to be agreed on early and held to exactly.

What's next

• Collision-aware planning — the current Cartesian decomposition assumes a clear path; a proper motion planner (RRT or the Kortex built-in planner) would handle obstacles without manual step tuning
• Grasp verification — add a force/torque threshold check after the gripper closes to confirm the brick was actually caught before attempting the place move
• Orientation interpolation — linear Euler interpolation can produce unexpected wrist trajectories; quaternion SLERP would give cleaner rotational paths
• 3D brick pile — the reserve bricks are currently laid flat in a single layer; handling a stacked or random pile would require depth estimation and a more general grasp planner