← Back to Blog

Two Macs, One Queue

Listen to this postRead by Nat · 5:37

I had a long-running experiment on my MacBook and a Mac Mini doing absolutely nothing ten feet away. This seemed like a lousy arrangement. Or maybe I was just trying to justify buying the Mac Mini to myself. I know there are dedicated solutions for this, but I wanted to do it myself for fun.

I wondered how hard it would be to make the Mini useful without becoming a cluster administrator, and I had no interest in running Kubernetes at home or paying a huge cloud bill.

The old version of the experiment ran for 3 hours and 20 minutes on my MacBook without finishing. My first run using both Macs finished in 16 minutes and 58 seconds. Much less. Now, it's not a perfect comparison because the experiment changed slightly, but that was enough to make me realize that the Mac sitting there could definitely speed things up.

A cluster, technically

The MacBook is in charge. It starts the run, does some of the work itself, records completed jobs, and combines the results. The Mac Mini accepts jobs over SSH and sends the output files back. At some point I'd like to make it so the system is agnostic of which machine is in charge, but it's not there yet.

                         private Git repository
                         code and job definitions
                                  |
                                  v
MacBook coordinator  ------- SSH ------->  Mac Mini worker
  assigns jobs                              runs jobs
  runs jobs too                             returns files
  records progress  <--------------------  reports completion

The diagram makes this look more official than it actually is. Git keeps the code straight, Remote Login lets the MacBook run commands on the Mini over SSH, and a small coordinator hands each job to the next free worker.

This only works when the work comes apart cleanly. Simulations, model evaluations, renders, data processing, and parameter sweeps are good candidates because each job can take one slice of the input and write one output file. If job 19 needs the result of job 18, another computer will spend a lot of time waiting. So parallelization is really the name of the game.

I started with both Macs on the same network and an account I controlled on each one. On the worker, open System Settings → General → Sharing, turn on Remote Login, and limit access to the account that will run the jobs.

Open Terminal on the worker and check its short username:

whoami

From the coordinator, try logging in:

ssh worker-user@worker-mac.local

If the .local name does not work, find the worker's local IP address under System Settings → Network and try that:

ssh worker-user@192.168.x.x

Use the username from the worker, not the coordinator. I first tried mine. The Mini immediately closed the connection because, quite reasonably, that account did not exist. It took me longer than I would like to admit to notice this.

For unattended jobs, make a dedicated SSH key on the coordinator and copy the public key to the worker:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_compute_cluster
ssh-copy-id \
  -i ~/.ssh/id_ed25519_compute_cluster.pub \
  worker-user@worker-mac.local

The private key stays on the coordinator, while the worker gets only the line from the .pub file. Neither belongs in Git. I also added a short name to ~/.ssh/config because I didn't want usernames and addresses scattered through my scripts:

Host compute-mini
    HostName worker-mac.local
    User worker-user
    IdentityFile ~/.ssh/id_ed25519_compute_cluster
    IdentitiesOnly yes

Once both machines had the same checkout at ~/src/my-project, I could prove the whole idea from two terminals:

# First terminal
cd ~/src/my-project
python run_experiment.py --start 0 --stop 500 --output results/local.json

# Second terminal
ssh compute-mini \
  'cd ~/src/my-project && python run_experiment.py --start 500 --stop 1000 --output results/remote.json'

# When the remote job finishes
scp compute-mini:~/src/my-project/results/remote.json results/

At this point both Macs can work at the same time, although calling this arrangement a cluster is still a little bit generous.

SSH was the easy part

I expected SSH to be the annoying part, but once it worked I still had remote access rather than parallel computing. The experiment was one large command, so the two machines couldn't share it and a failure near the end still left me with nothing.

I changed the command so each job owned a small range of inputs and one output file:

python run_experiment.py \
  --start 0 \
  --stop 100 \
  --output results/shard-000.json

Another process can handle inputs 100 through 199 at the same time. Neither process needs to know the other exists. A final command combines the shard files when they are all done.

Small shards also make failure cheap. If one process stops, I rerun its hundred inputs instead of the whole experiment. The output file is the receipt. The coordinator accepts a job as complete only after the file has finished writing and its hash matches the ledger.

That only works if every job follows a few rules:

  • Give every job its own output path.
  • Write to a temporary file, then rename it when the write completes.
  • Make the same job produce the same result when possible.
  • Keep the final merge separate from the jobs.

I also pin every run to a full Git commit:

git rev-parse HEAD

The coordinator checks out that commit on each machine and installs dependencies from the lockfile. Large inputs don't go in Git, but the job records a SHA-256 hash for each one and the worker verifies it after the file arrives.

Two unequal computers

My first instinct was to split the jobs in half. That would have been a mistake. The Mini was much faster than the MacBook at this workload, so it would have finished its half and then sat idle while the MacBook kept grinding.

Instead, my coordinator reads the job list and gives each Mac a fixed number of worker slots. Whenever a slot finishes, it gets the next job from one shared queue.

The toy scheduler below shrinks the experiment to 36 jobs. Try the even split first. The Mini races through its half and then has nothing to do. The shared queue fixes that. While it is running, restart the MacBook and watch what the ledger saves.

The times are simplified, but the four MacBook slots, eight Mini slots, and lopsided job count match the real setup.

A useful accident

While I was doing this, both Activity Monitors lit up, and this was very satisfying, but it didn't prove much. So I stopped the coordinator in the middle of a run.

This caused me to find a real bug almost immediately. Closing an SSH connection didn't always kill the process it had started on the Mini. The coordinator would restart, see an unfinished job, and launch it again while the old copy was still running. More workers had created a new and exciting way to waste twice the compute.

I fixed this by recording each remote process group. On restart, the coordinator stops stale groups before it hands out any unfinished jobs. It also checks every output hash against the ledger. A valid file stays done. A missing or half-written file goes back in the queue.

Then, by total accident, my MacBook restarted for real while the Mini stayed on. When the MacBook came back, the coordinator cleaned up the old remote work and continued from the last finished shard. I had planned a tidy recovery test and got a messier, better one.

The computer was already there

This was the right amount of infrastructure for trusted work on two computers I already own. If I keep it running, the Mini gets its own non-administrator account. SSH password login stays off, and port 22 stays off the public internet. Jobs that need a GPU or a hundred machines for an hour still belong in the cloud.