I enjoy the challenge of hacking physics engines and sandbox games to do elaborate, absurd, and seemingly impossible things. When DoodleChaos released this video of marbles unconvincingly falling through a pegboard playing the piano, I saw another challenge to take on.
His technique was clever but left some viewers unsatisfied.
Can we do better? “Order from Chaos” is statistically impossible. We can’t expect randomly moving things to fall in order. Probabilistically, marbles falling through a pegboard should end up looking like a bell curve.
From a thermodynamics perspective, we can interpret randomness as entropy, where low entropy is order and high entropy is chaos. In a few words, the second law of thermodynamics states that entropy always increases. It’s a big part of how we perceive time as moving forward: when perturbed, ordered things tend to fall apart, and disorganized things remain in disarray. In order to make order from chaos, we need to reverse entropy (a plot device from a sci-fi classic). It’s basically impossible to find the initial conditions that result in spontaneous order. But we can break the rules if we make our own physics.
“If you wish to reverse entropy, you must first reinvent the universe” — Carl Sagan (not really)
We want to make these marbles strike these piano keys at these times. These are our final conditions, we just have to simulate physics backwards from there. This requires us to build a custom physics engine with that capability. It’s like building our own laws of physics, inventing our own universe that operates under our own rules, a universe where we can reverse entropy.
Prior work
First thing I did was research prior work on reverse physics simulation. My alma mater did this for the general case:
The results look kinda impressive but also look like they are being played backwards, due to how we perceive increasing entropy as the arrow of time:
when reverse-simulation motion is played back, we see a collection of objects go from being totally disordered to completely organized. This looks unnatural to us because in the real world (loosely speaking) entropy should increase as time advances.
The paper describes that the biggest problem to solve with reverse physics simulation is how to handle dissipative forces like collisions and friction.
If the forces are autonomous and independent of velocity, such as for a bunch of particles interacting via gravity, then the forces will look the same forward or backward in time, and the system can be said to have time-reversal symmetry.
However, for other systems, such as those involving velocity-dependent damping forces, the dynamics are not necessarily symmetric, e.g., if energy is lost moving forward in time, then it is gained moving backward in time. While the extension to rigid body animation might seem straightforward, unfortunately, as they say, the “devil is in the details,” and running systems backward in time is not for the faint of heart.
As long as energy is conserved, physics will behave mostly the same forwards and backwards. However, if there is energy loss, like when a marble bounces off a peg and the peg absorbs some of the marble’s kinetic energy, simulating in reverse will cause the marble to gain energy when bouncing. And, since the marble will be moving faster and will quickly collide with another peg, it’ll gain more and more energy until it launches itself out of the simulation.
The techniques described in the paper are rather complicated since they handle the general case of arbitrary 3D objects colliding and sliding around. However, all I have are a bunch of frictionless 2D balls, so these techniques are overpowered and not worth implementing. I’m better off rolling something of my own, though their technique of manipulating collisions gives me a clue.
Prerequisite: Inventing the Universe
I threw together a simple physics engine in Javascript to see how balls would behave running down a pegboard, and how they would behave backwards.
As predicted, the backwards simulation explodes, and none of the balls make it to the top.
Method 1: Non-colliding samples
As we saw with naively throwing marbles at the pegboard from the bottom, we can’t expect any of them to follow a path that makes sense forwards. In our universe, we do not want marbles teleporting into the pegboard. We need to design a viable path for the marble to travel up, aiming the energy it gains from each bounce precisely upwards so that gravity can dissipate that energy. There’s an easy way that doesn’t require much designing: just sample forward-simulated paths.
Given a pegboard and piano keys:
- Pick a random initial condition (start position and velocity) above the pegboard for a single marble
- Run the simulation until the marble lands on a key
- Record the path the marble took
- Go to step 1 until you have paths of marbles hitting all the keys you need
From there, we’ll have a collection of paths that we can sample from and arrange into marbles hitting piano keys in time with music.
And it turns out it’s already been done, though not with a pegboard but the method is the same.
The biggest limitation of this method is that marbles must not collide with each other. There are ways to avoid collisions for simulations more dense in marbles, but I think ball-ball collisions are where it gets chaotic and fun. We can do better.
Method 2: Sample splicing
We can synthesize viable paths that handle ball-ball collisions by splicing together forward-simulated “lookup” paths with “on-demand” reverse simulation that would normally cause explosions.
First, let’s expand upon Method 1. Instead of recording paths as start to end, we take advantage of the repeating pattern of the pegboard and record paths as segments between peg collisions, generating a lookup of viable paths.
- Simulate a ball falling through an infinite pegboard, starting from the peg at the origin.
- When the ball collides with a peg, remember the offset between the ball and the peg and move the ball back to the peg at the origin with the same offset. It will fall the exact same way because the pegboard is the same no matter where the ball is.
- Record the paths of the ball in between collisions and record the offset and exit velocity of each collision. Put this data into the viable path lookup.
When we have enough samples in the viable path lookup, we can pick a viable path for the ball by looking up our end condition (a collision with a peg with an offset and velocity) and picking the path that ends with the most similar collision. The collision won’t have the exact same offset and velocity as our specified end condition, but our eyes probably won’t be able to tell (borrowing from the previous paper’s technique of manipulating collisions).
Next, we handle ball-ball collisions and put it all together:
- Start with a collision with a peg that results in the ball free-falling (not colliding with anything) and striking the right piano key at the right time,
- Pick a viable path that ends with the most similar collision and run backwards until there is a ball-ball collision.
- On ball-ball collision, switch both balls into “on-demand” reverse simulation, as in simulate the physics mathematically backwards instead of using a lookup. We can avoid ball-ball collisions from exploding by making the balls have perfectly elastic collisions that conserve energy.
- On ball-peg collision where the ball is in “on-demand” reverse simulation, switch to the viable path lookup and pick the closest viable path.
- There are conditions where this fails, like when a ball is blocked from going from a piano key to the peg board, in which case pick a different viable path until it works. Otherwise, due to the viable path lookup, we are guaranteed to have the balls eventually go back up.
With this algorithm of switching back and forth between reverse simulation and viable path lookup (and a lot of other adjustments and optimizations to handle limitations and other features needed for specific use cases like how to aim for the piano keys), we effectively synthesize our own plausible-looking physics by splicing together pre-simulated paths with paths generated from reverse simulation, and from there we can force marbles to appear to do the impossible.
Results
We can make the marbles defy the normal distribution.
We can make the marbles spontaneously assemble.
And, finally, we can make the marbles play music.
Other Details
The pitches of the piano key and marbles are colored according to the circle of fifths. Similar colors represent pitches more consonant with each other. Details here:
Going with the theme of physical modeling, where marbles collide with things that sound like a material, I thought it would feel more physical if the piano keys sounded like a percussion instrument being struck, since it doesn’t really make sense for marbles to strike actual piano keys which need to be held down to sound. I used the Chromaphone synth to make it sound like a vibraphone.
Code for the Giant Steps video available here as a p5.js sketch:
Physical Modeling Sound Synthesis
A viewer of DoodleChaos’s video noted the sound of the marbles.
The sound was stock audio, but I thought we could make even more satisfying sounds by generating audio based on the simulation so collisions are actually audible.
First, we record all collisions. The simulation frequency is set really high (960Hz, or 16 steps per video frame) so the details of rapid bouncing can be heard. Then, we encode the collisions as impulses in a 44.1kHz waveform to be exported as an audio file (this was simpler than encoding as MIDI, which had issues with resolution in time and amplitude). We need to randomly spread out the impulses (dithering in time) or else we’ll get a strange buzzing at 960Hz.
Then we design an audio processor that converts the impulses into the sound of materials colliding. The most important parts are the noise burst and the resonator. The noise burst represents the collision itself, approximating how hard the materials are. The resonator represents how the material vibrates in response, giving a sonic impression of what the material is, e.g. wood, metal, plastic. Beyond that, the details of sound design are more of an art than science.
This is what it sounds like by itself. Enjoy.
For more physics and music hackery, follow me on Twitter
