Thin-shell Object Manipulations with Differentiable Physics Simulations

Under Double-Blind Review

Abstract

In this work, we aim to teach robots to manipulate various thin-shell materials. we introduce ThinShellLab - a fully differentiable simulation platform tailored for robotic interactions with diverse thin-shell materials possessing varying material properties, enabling flexible thin-shell manipulation skill learning and evaluation. This comprehensive endeavor encompasses a triad of experiment task categories, which are as follows: manipulation tasks, inverse design tasks, and real-world experiments. Through these diverse task categories, we aim to comprehensively explore and harness the potential of ThinShellLab, advancing the field of robotics and thin-shell material manipulation.

Manipulation Tasks

The manipulation tasks entail the use of manipulators constructed from hyper-elastic materials, which interact with thin-shell materials through physical contact. In all of these tasks, the primary objective is to devise an optimal action sequence for the manipulators to attain the highest possible rewards.

Balancing

Scene Setting

This experiment employs two grippers to hold a paper sheet on both side with a small ball on it.

Objective

The primary objective of the agent is to ensure the ball remains consistently positioned at the center of the sheet.

Throwing

Scene Setting

Similar to Balancing, this task also entails two grippers delicately gripping a sheet of paper, with a small ball positioned atop it.

Objective

The objective is to propel the ball to attain maximum vertical height.

Forming

Scene Setting

It begins with a curved paper on the table, one side fixed, and a gripper on top.

Objective

The agent's objective is to manipulate the paper using a single manipulator to achieve a predefined goal shape. This goal shape is generated by executing a random trajectory and saving the final paper position.

Following

Scene Setting

There's a sheet of paper on a table, a cube resting on it, and a parallel gripper holding the paper.

Objective

his task involves moving the cube along with the paper towards the right side as far as possible by pulling the paper on an appropriate speed.

Separating

Scene Setting

Identical to Following

Objective

The agent need to pull out the paper and separate it from the cube horizontally, striving for maximum separation distance.

Separating-Cloth

Scene Setting

In this particular task, we employ a cloth material characterized by low bending stiffness, making it unfeasible to impart an initial speed through pushing, thus introducing a material-dependent variation in behavior.

Objective

Identical to Separating .

Pick-Folding

Scene Setting

In this task, the agent operates two manipulators initialized on opposite sides of a sheet of paper resting on an arched table.

Objective

The objective is to pick up the paper and fold it, with the specific aim of creating a crease in the middle of the paper.

Folding-Lower

Scene Setting

This task revolves around folding a sheet of paper to induce plastic deformation. The scene comprises a bending strip on the table with one end fixed and a manipulator positioned on it.

Objective

The objective is to reinforce the lower curve.

Folding-Upper

Scene Setting

Identical to Folding-Lower.

Objective

The objective for this task is to reinforce upper curve.

Lifting

Scene Setting

This task begins with a flat sheet with two manipulators on the bottom, one on the top, and a block positioned on the opposite side.

Objective

The objective is to raise the block using the sheet and transport it to a different location.

Inverse Design Tasks

The Inverse Design tasks show the differentiability of ThinShellLab can be used for optimizaing the scene parameters. In this set of tasks, we maintain fixed manipulator trajectories and instead optimize the scene parameters to maximize specific objectives.

Bouncing

We begin with a flat sheet on the table featuring two creases. The sheet initially bounces due to the bending force exerted by the creases. The task's objective is to optimize the bending stiffness to achieve higher bounce height.

Initial

Optimized

Sliding

The scene consists of three sheets of paper stacked on a table, with a manipulator pressing and sliding on them. The goal is to optimize the friction coefficient between the sheets to ensure they all move together.

Initial

Optimized

Card

Initialization involves a card held by three manipulators, followed by executing a card shuffling movement to launch the card. The aim in this task is to adjust the bending stiffness to project the card as far to the right as possible. It's a delicate task, so we show the zoom-in view of the last frame to show the x-axis position of the card.

Initial

Optimized

Real-World Experiments

We demonstrate how ThinShellLab can help bridge simulation and reality by several real-world experiments. These experiments include real-to-sim system identification and sim-to-real skill transferring.

Real-to-Sim System identification

To reconstruct real scenes in simulation, we first collect real-world data by vision-based tactile sensors and force sensors. As in the Inverse Design tasks, we set up differentiable objectives measuring the distance between simulation and the real scene. The system identification is then solved by optimizing the scene parameters.

Real Scene

Simulation Scene

Real Sensor Data

Optimized Sensor Data

Sim-to-Real Skill Transferring

We show how a learned skill from the Separating task can be deployed to a real robot system. We first recover the scene parameters including material parameters and friction coefficients from collected data, and then train a policy in the reconstructed scene. As shown below, the policy can be successfully transfered to reality.

Separating-Sim

Separating-Real

In comparison, we also demonstrate a straightforward heuristic policy of dragging at constant speed. Its speed is equal to the maximal speed in the leared policy. This policy failed to separate the object and the paper due to insufficient relative speed.

Separating-Heuristic