Building an Affordable High-Performance Inference Machine Using Tesla V100 GPUs
In the evolving world of machine learning and AI development, affordability and performance are key considerations for enthusiasts and professionals alike. Recently, I embarked on constructing a cost-effective inference setup that balances these factors, culminating in a practical system equipped with three Tesla V100 GPUs. Here’s a comprehensive overview of the project, including hardware assembly, cooling solutions, software configurations, and performance benchmarks.
Hardware Selection and Cost Breakdown
The core of this build consists of three Tesla V100 GPUs, chosen for their robust inference capabilities. Originally, I intended to acquire three Tesla P100 units; however, due to a purchasing mishap, I received three V100s instead—an unintended but fortunate surprise. The total investment for this setup was approximately $1,500.
To support these high-performance GPUs, I included 64GB of ECC DDR4 RAM, ensuring reliable memory operations during intensive inference tasks. The entire system is designed to operate headlessly, remotely managed over a network.
Innovative Cooling Solution
Maintaining optimal operating temperatures is critical when deploying multiple GPUs in a confined space. To this end, I crafted a custom cooling block using 3D printing technology. The cooling block snugly fits into the rectangular opening at the back of the case, with minor manual adjustments using a file to achieve a precise friction fit. It houses three server-grade fans that actively pull air from inside the case through the GPUs to the exterior, significantly reducing GPU temperatures during inference runs.
Temperature management is effective; GPU temperatures stay below 55°C even under load, though the setup can be somewhat noisy. This balance of cooling efficiency and operational noise was acceptable given the cost savings.
Remote Management and Software Optimization
For remote control and system management, I utilized Tailscale, enabling seamless headless operation from my Mac. Additionally, GPT Astra running on Codex allows me to remotely access and control the machine, facilitating various testing and optimization tasks.
To maximize inference performance, I conducted numerous experiments tuning different configuration parameters. This included adjusting metrics like maximum token predictions (mtp), implementing speculative decoding strategies, optimizing GPU placement, and configuring prefix caching—all aimed at enhancing prefill and decode speeds.
Performance Benchmarks
The system was tested with several prominent models:
-
Qwen 3.8 27B: With prefill speeds around 600-700 tokens per second (tps) and decode speeds approximately 30-40 tps under standard conditions. When processing complex tasks with a 256k context window, decode throughput slightly decreased to around 20 tps.
-
Qwen 3.8 Flash (Next Model): Achieved prefill rates of roughly 90 tokens/sec and decode speeds near 20 tokens/sec during intensive inference.
Overall, I am pleased with the performance-to-cost ratio of this build. The Tesla V100s, despite their age, remain viable and economical options for AI inference workloads—though setting them up can be somewhat challenging.
Conclusion
This project demonstrates that with thoughtful engineering and innovative cooling solutions, building a potent inference machine on a modest budget is achievable. For those interested in cost-efficient AI deployment, Tesla V100 GPUs present a compelling option, provided one is prepared for the setup complexity.
If you’re considering a similar configuration, ensure reliable remote management capabilities and be ready to experiment with software optimizations to get the best out of your hardware.
Feel free to reach out with questions or share your own experiences working with similar setups.
