Two Macs, One Queue

September 2, 20266 min read

I had a long-running experiment on my MacBook and a Mac Mini doing absolutely nothing ten feet away. This seemed like a lousy arrangement.

I wondered how hard it would be to make the Mini useful without becoming a cluster administrator in my own home. I had no interest in running Kubernetes at home or paying a cloud bill while an idle computer sat ten feet away.

The old version of the experiment ran for 3 hours and 20 minutes on my MacBook without finishing. My first run using both Macs finished in 16 minutes and 58 seconds, including the time I spent stopping and restarting it to see what would break.

I had changed the program too, so it would be dishonest to say the Mini made it 12 times faster. I wanted to know how much of the speedup the second Mac had actually earned.

A cluster, technically

The MacBook is in charge. It starts the run, does some of the work itself, records completed jobs, and combines the results. The Mac Mini accepts jobs over SSH and sends the output files back.

                         private Git repository
                         code and job definitions
                                  |
                                  v
MacBook coordinator  ------- SSH ------->  Mac Mini worker
  assigns jobs                              runs jobs
  runs jobs too                             returns files
  records progress  <--------------------  reports completion

The diagram makes this look more official than it is. Git keeps the code straight, Remote Login lets the MacBook run commands on the Mini over SSH, and a small coordinator hands each job to the next free worker.

This only works when the work comes apart cleanly. Simulations, model evaluations, renders, data processing, and parameter sweeps are good candidates because each job can take one slice of the input and write one output file. If job 19 needs the result of job 18, another computer will spend a lot of time waiting.

I started with both Macs on the same network and an account I controlled on each one. On the worker, open System Settings → General → Sharing, turn on Remote Login, and limit access to the account that will run the jobs.

Open Terminal on the worker and check its short username:

whoami

From the coordinator, try logging in:

ssh worker-user@worker-mac.local

If the .local name does not work, find the worker's local IP address under System Settings → Network and try that:

ssh worker-user@192.168.x.x

Use the username from the worker, not the coordinator. I first tried mine. The Mini immediately closed the connection because, quite reasonably, that account did not exist. It took me longer than I would like to admit to notice.

For unattended jobs, make a dedicated SSH key on the coordinator and copy the public key to the worker:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_compute_cluster
ssh-copy-id \
  -i ~/.ssh/id_ed25519_compute_cluster.pub \
  worker-user@worker-mac.local

The private key stays on the coordinator, while the worker gets only the line from the .pub file. Neither belongs in Git. I also added a short name to ~/.ssh/config because I didn't want usernames and addresses scattered through my scripts:

Host compute-mini
    HostName worker-mac.local
    User worker-user
    IdentityFile ~/.ssh/id_ed25519_compute_cluster
    IdentitiesOnly yes

Once both machines had the same checkout at ~/src/my-project, I could prove the whole idea from two terminals:

# First terminal
cd ~/src/my-project
python run_experiment.py --start 0 --stop 500 --output results/local.json

# Second terminal
ssh compute-mini \
  'cd ~/src/my-project && python run_experiment.py --start 500 --stop 1000 --output results/remote.json'

# When the remote job finishes
scp compute-mini:~/src/my-project/results/remote.json results/

At this point both Macs can work at once, although calling the arrangement a cluster is still generous.

SSH was the easy part

I expected SSH to be the annoying part, but once it worked I still had remote access rather than parallel computing. The experiment was one large command, so the two machines couldn't share it and a failure near the end still left me with nothing.

I changed the command so each job owned a small range of inputs and one output file:

python run_experiment.py \
  --start 0 \
  --stop 100 \
  --output results/shard-000.json

Another process can handle inputs 100 through 199 at the same time. Neither process needs to know the other exists. A final command combines the shard files when they are all done.

Small shards also make failure cheap. If one process stops, I rerun its hundred inputs instead of the whole experiment. The output file is the receipt. The coordinator accepts a job as complete only after the file has finished writing and its hash matches the ledger.

That only works if every job follows a few rules:

Give every job its own output path.
Write to a temporary file, then rename it when the write completes.
Make the same job produce the same result when possible.
Keep the final merge separate from the jobs.

I also pin every run to a full Git commit:

git rev-parse HEAD

The coordinator checks out that commit on each machine and installs dependencies from the lockfile. Large inputs don't go in Git, but the job records a SHA-256 hash for each one and the worker verifies it after the file arrives.

This sounded a little precious when I built it. Then I pictured finding a surprising result three days later and having no idea which version of the code produced it. A branch name moves. latest is worse. The commit doesn't move.

I record the command, host, input hashes, output hash, and runtime for the same reason. Those details are boring right up until I distrust an answer.

Two unequal computers

My first instinct was to split the jobs in half. That would have been a mistake. The Mini was much faster than the MacBook at this workload, so it would have finished its half and then sat idle while the MacBook kept grinding.

Instead, my coordinator reads the job list and gives each Mac a fixed number of worker slots. Whenever a slot finishes, it gets the next job from one shared queue. The Mini kept asking for more work and completed 194 of the 216 jobs.

The toy scheduler below shrinks the experiment to 36 jobs. Try the even split first. The Mini races through its half and then has nothing to do. The shared queue fixes that. While it is running, restart the MacBook and watch what the ledger saves.

The times are simplified, but the four MacBook slots, eight Mini slots, and lopsided job count match the real setup.

A useful accident

Both Activity Monitors lit up. That was satisfying, but it didn't prove much. So I stopped the coordinator in the middle of a run.

I found a real bug almost immediately. Closing an SSH connection didn't always kill the process it had started on the Mini. The coordinator would restart, see an unfinished job, and launch it again while the old copy was still running. More workers had created a new and exciting way to waste twice the compute.

I fixed this by recording each remote process group. On restart, the coordinator stops stale groups before it hands out any unfinished jobs. It also checks every output hash against the ledger. A valid file stays done. A missing or half-written file goes back in the queue.

Then my MacBook restarted for real. The Mini stayed on. When the MacBook came back, the coordinator cleaned up the old remote work and continued from the last finished shard. I had planned a tidy recovery test and got a messier, better one.

The full system was at least 11.8 times faster, but the Mini doesn't deserve all the credit because I had changed the program too. Comparing the old three-slot runner with the new twelve-slot run puts the gain from using both Macs closer to 2.9 times. Three hours makes every idea feel expensive. Seventeen minutes makes it easy to try something dumb and find out.

The computer was already there

This was the right amount of infrastructure for trusted work on two computers I already owned. If I keep it running, the Mini gets its own non-administrator account, SSH password login stays off, and port 22 stays off the public internet. Jobs that need a GPU or a hundred machines for an hour still belong in the cloud.

The Mini handled 194 of the 216 jobs in that first run. I bought it for ordinary computer things. Apparently it had been waiting for a promotion.