Article Summary

Apple discontinued support for non-Apple hardware graphics controllers when moving to Apple Silicon ARM-based processors. This has limited Apple systems to utilizing only the internal processing power when running local LLMs and “AI” models.  However tinygrad has stepped in to fill the gap with TinyGPU - an app that enables both AMD and NVIDIA eGPUs over USB4/Thunderbolt.

Running local LLMs or image generation on a MacBook or Mac Mini’s internal hardware can cause reduced performance, input lag, or even freezing and disconnections in streaming meetings. This is caused by the LLM taking up the system’s internal resources. Until TinyGPU the only option has been to disable the local model or use a second computer for running the model, but TinyGPU finally restores eGPU support ( for GPU compute - not external displays ) to Apple Silicon Macs.

Why is 3rd party software necessary?

Apple removed support for external GPUs when moving from Intel to Apple M-Series processors. Without the underlying driver support, the cards are detected, but they are incapable of performing any tasks.

As eGPUs have moved from a slick gaming accessory to a productivity and LLM enhancement, the need has suddenly grown for better eGPU support with Apple systems to bring them in line with Windows hosts.

What is tinygrad?

At its core, tinygrad serves as a comprehensive, end-to-end deep learning stack designed for performance.  It provides similar functionality to PyTorch, JAX, and TVM, but is fully hackable and is available from Github 

TinyGPU.app is a driver extension to restore eGPU compute functionality to M-Series Macs.

Installation

TinyGPU installation instructions are provided in the tinygrad documentation and may be subject to change.  In short:

  1. Connect the eGPU to the Mac via USB4/Thunderbolt
  2. Run the setup script from the tinygrad documentation
  3. Enable the driver extension through Apple’s System Settings
  4. Install the compiler for either AMD or NVIDIA graphics hardware

First impressions and pain points

We initially tested TinyGPU with an AMD Radeon 9070 and NVIDIA RTX 5070 in our TBT5-AI Thunderbolt 5 External GPU Enclosure with a 13-inch MacBook Pro with M2 processor.

TinyGPU setup completed successfully, and we used the tinygrad local LLM demo for basic testing.

In general, the AMD graphics performed well, but there wasn’t much of a performance boost over the baseline system performance.

NVIDIA requires installing Docker Desktop, which adds to the setup and possibly bottlenecks some performance. Again, we didn’t see much improvement over the baseline system performance.

We think this is a great project to keep an eye on, and look forward to seeing how eGPU performance and compatibility measure up with Apple Silicon host systems.

カテゴリ内の他の記事を見る

LLM, macOS

関連記事

Loading Comments

Article ID: 746971758823