[ad_1]
If you are flexible about the GPU model and want to choose the most cost-effective cloud GPU, check out our cloud GPU $ / throughput chart.
If you know the specific cloud GPU (e.g. A100) that you want, identify the cloud providers that offer it.
Cloud GPU price per throughput
Spot GPUs
| Cloud | GPU Type / Memory | # of GPUs | Spot | Throughput* | Throughput* / $*** |
|---|---|---|---|---|---|
| Jarvislabs | A100 / 40 GB | 1 | 0.69 | 179 | 259 |
| GCP | A100 / 40 GB | 1 | 1.1 | 179 | 163 |
| Azure | A100 / 80 GB | 1 | 1.47 | 232 | 157 |
| Azure | A100 / 80 GB | 4 | 5.88 | 821 | 140 |
| Azure | A100 / 80 GB | 2 | 2.94 | 406 | 138 |
| GCP | V100 / 16 GB | 1 | 0.84 | 42 | 50 |
| GCP | V100 / 16 GB | 2 | 1.68 | 77 | 46 |
| AWS | V100 / 16 GB | 1 | 0.92 | 42 | 46 |
| GCP | V100 / 16 GB | 4 | 3.36 | 153 | 46 |
| GCP | V100 / 16 GB | 8 | 6.72 | 289 | 43 |
| AWS | V100 / 16 GB | 4 | 3.67 | 153 | 42 |
| AWS | V100 / 16 GB | 8 | 7.34 | 289 | 39 |
| Azure | V100 / 16 GB | 1 | 1.35 | 42 | 31 |
| Azure | V100 / 16 GB | 2 | 2.7 | 77 | 29 |
| Azure | V100 / 16 GB | 4 | 5.4 | 153 | 28 |
* Training throughput is a good metric to measure relative GPU effectiveness. It measures the number of tokens processed per second by the GPU for a language model (i.e. bert_base_squad).1 Please note that these throughput values should be considered in comparing different GPUs. The same hardware would have a significantly different throughput for your workload since there is significant throughput difference even between LLMs running on the same hardware.2
*** Excludes cost of storage, network performance, ingress/egress etc. This is only the GPU cost.3
In all these throughput per dollar tables:
- Not all possible configurations are listed, deep learning focused configurations that are similar to AWS’ options are included.
- West or Central US regions were used where possible
- These are the listed prices for each category, they can be improved
On-demand GPUs from big tech cloud providers
| Cloud | GPU Type / Memory | # of GPUs | On-demand $ | Throughput* | Throughput* / $*** |
|---|---|---|---|---|---|
| Azure | A100 / 80 GB | 1 | 3.67 | 232 | 63 |
| Azure | A100 / 80 GB | 4 | 14.69 | 821 | 56 |
| Azure | A100 / 80 GB | 2 | 7.35 | 406 | 55 |
| GCP | A100 / 40 GB | 1 | 3.67 | 179 | 49 |
| Azure | A100 / 80 GB | 8 | 37.18 | 1,362 | 37 |
| AWS | A100 / 80 GB | 8 | 40.97 | 1,362 | 33 |
| GCP | V100 / 16 GB | 1 | 2.95 | 42 | 14 |
| AWS | V100 / 16 GB | 1 | 3.06 | 42 | 14 |
| Azure | V100 / 16 GB | 1 | 3.06 | 42 | 14 |
| GCP | V100 / 16 GB | 2 | 5.91 | 77 | 13 |
| GCP | V100 / 16 GB | 4 | 11.81 | 153 | 13 |
| AWS | V100 / 16 GB | 4 | 12.24 | 153 | 13 |
| Azure | V100 / 16 GB | 2 | 6.12 | 77 | 13 |
| Azure | V100 / 16 GB | 4 | 12.24 | 153 | 13 |
| GCP | V100 / 16 GB | 8 | 23.63 | 289 | 12 |
| AWS | V100 / 16 GB | 8 | 24.48 | 289 | 12 |
On-demand GPUs from other cloud providers
| Cloud | GPU Type / Memory | # of GPUs | On-demand $ | Throughput* | Throughput* / $ |
|---|---|---|---|---|---|
| Jarvislabs | A100 / 40 GB | 1 | 1.1 | 179 | 163 |
| Lambda | A100 / 40 GB | 1 | 1.1 | 179 | 163 |
| Lambda | H100 / 80 GB | 1 | 1.99 | 322 | 162 |
| Datacrunch | A100 / 80 GB | 1 | 1.85 | 232 | 125 |
| Latitude.sh | H100 / 8 GB | 8 | 22.42 | 2,693 | 120 |
| Lambda | A100 / 80 GB | 8 | 12 | 1,362 | 114 |
| Datacrunch | A100 / 80 GB | 4 | 7.4 | 821 | 111 |
| Datacrunch | A100 / 80 GB | 2 | 3.7 | 406 | 110 |
| Latitude.sh | H100 / 8 GB | 4 | 11.96 | 1,321 | 110 |
| Datacrunch | A100 / 80 GB | 8 | 14.8 | 1,362 | 92 |
| Oblivus Cloud | A100 / 80 GB | 1 | 2.55 | 232 | 91 |
| RunPod | A100 / 80 GB | 8 | 15.12 | 1,362 | 90 |
| Oblivus Cloud | A100 / 80 GB | 2 | 5.1 | 406 | 80 |
| Oblivus Cloud | A100 / 80 GB | 4 | 10.2 | 821 | 80 |
| Oblivus Cloud | A100 / 40 GB | 1 | 2.39 | 179 | 75 |
| Paperspace | A100 / 80 GB | 1 | 3.18 | 232 | 73 |
| Oblivus Cloud | A100 / 80 GB | 8 | 20.4 | 1,362 | 67 |
| Lambda | V100 / 16 GB | 8 | 4.4 | 289 | 66 |
| Oblivus Cloud | V100 / 16 GB | 1 | 0.65 | 42 | 65 |
| Paperspace | A100 / 80 GB | 4 | 12.72 | 821 | 65 |
| Paperspace | A100 / 80 GB | 2 | 6.36 | 406 | 64 |
| Oblivus Cloud | V100 / 16 GB | 2 | 1.3 | 77 | 59 |
| Oblivus Cloud | V100 / 16 GB | 4 | 2.6 | 153 | 59 |
| Paperspace | A100 / 40 GB | 1 | 3.09 | 179 | 58 |
| Paperspace | A100 / 80 GB | 8 | 25.44 | 1,362 | 54 |
| Exoscale | V100 / 16 GB | 4 | 3.32 | 153 | 46 |
| Datacrunch | V100 / 16 GB | 1 | 1 | 42 | 42 |
| Datacrunch | V100 / 16 GB | 2 | 2 | 77 | 39 |
| Datacrunch | V100 / 16 GB | 4 | 4 | 153 | 38 |
| Exoscale | V100 / 16 GB | 2 | 2.01 | 77 | 38 |
| Datacrunch | V100 / 16 GB | 8 | 8 | 289 | 36 |
| Exoscale | V100 / 16 GB | 1 | 1.38 | 42 | 30 |
| OVHcloud | V100 / 16 GB | 1 | 1.97 | 42 | 21 |
| OVHcloud | V100 / 16 GB | 2 | 3.94 | 77 | 20 |
| OVHcloud | V100 / 16 GB | 4 | 7.89 | 153 | 19 |
| Paperspace | V100 / 16 GB | 1 | 2.3 | 42 | 18 |
Cloud GPU availability in different clouds
Input the model that you want in the search box to identify all cloud providers that offer it:
| Provider | GPU | Multi-GPU | On-demand $ / single GPU hour*** |
|---|---|---|---|
| AWS | M60 8 GB | 1, 2, 4x | $1.14 |
| AWS | T4 16 GB | 1, 2, 4, 8x | $1.20 |
| AWS | A10G 24 GB | 1, 4, 8x | $1.62 |
| AWS | V100 16 GB | 1, 4, 8x | $3.06 |
| AWS | V100 32 GB | 8x | $3.90** |
| AWS | A100 40 GB | 8x | $4.10** |
| AWS | A100 80 GB | 8x | $5.12** |
| CoreWeave | Quadro RTX 4000 8 GB | 1, 2, 3, 4, 5, 6, 7x | $0.35 |
| CoreWeave | RTX A4000 16 GB | 1, 2, 3, 4, 5, 6, 7x | $0.75 |
| CoreWeave | Quadro RTX 5000 16 GB | 1, 2, 3, 4x | $0.95 |
| CoreWeave | V100 16 GB | 1, 2, 3, 4, 5, 6, 7, 8x | $1.00 |
| CoreWeave | RTX A5000 24 GB | 1, 2, 3, 4x | $1.01 |
| CoreWeave | RTX A40 48 GB | 1, 2, 3, 4, 5, 6, 7, 8x | $1.68 |
| CoreWeave | RTX A6000 48 GB | 1, 2, 3, 4, 5, 6, 7, 8x | $1.68 |
| CoreWeave | A100 40 GB | 1, 2, 3, 4, 5, 6, 7, 8x | $2.46 |
| CoreWeave | A100 80 GB | 1, 2, 3, 4, 5, 6, 7, 8x | $2.61 |
| GCP | T4 16 GB | 1, 2, 4x | $0.75 |
| GCP | K80 12 GB | 1, 2, 4, 8x | $0.85 |
| GCP | P4 8 GB | 1, 2, 4x | $1.00 |
| GCP | P100 16 GB | 1, 2, 4x | $1.86 |
| GCP | V100 16 GB | 1, 2, 4, 8x | $2.88 |
| GCP | A100 40 GB | 1, 2, 4, 8, 16x | $3.67 |
| Jarvis Labs | Quadro RTX 5000 16 GB | 1x | $0.49 |
| Jarvis Labs | Quadro RTX 6000 24 GB | 1x | $0.99 |
| Jarvis Labs | RTX A5000 24 GB | 1x | $1.29 |
| Jarvis Labs | RTX A6000 48 GB | 1x | $1.79 |
| Jarvis Labs | A100 40 GB | 1x | $2.39 |
| Lambda Labs | Quadro RTX 6000 24 GB | 1, 2, 4x | $1.25 |
| Lambda Labs | RTX A6000 48 GB | 1, 2, 4x | $1.45 |
| Lambda Labs | V100 16 GB | 8x | $6.80 |
| Linode | Quadro RTX 6000 24 GB | 1, 2, 4x | $1.50 |
| Azure | K80 12 GB | 1, 2, 4x | $0.90 |
| Azure | T4 16 GB | 1, 4x | $1.20 |
| Azure | P40 24 GB | 1, 2, 4x | $2.07 |
| Azure | P100 16 GB | 1, 2, 4x | $2.07 |
| Azure | V100 32 GB | 8x | $2.75 |
| Azure | V100 16 GB | 1, 2, 4x | $3.06 |
| Azure | A100 40 GB | 8x | $3.40** |
| Azure | A100 80 GB | 1, 2, 4x | $3.67 |
| Azure | A100 80 GB | 8x | $4.096** |
| OVH | V100 16 GB | 1, 2, 4x | $1.79 |
| OVH | V100 32 GB | 1, 2, 4x | $1.99 |
| Paperspace | Quadro M4000 8 GB | 1x | $0.45 |
| Paperspace | Quadro P4000 8 GB | 1, 2, 4x | $0.51 |
| Paperspace | Quadro RTX 4000 8 GB | 1, 2, 4x | $0.56 |
| Paperspace | RTX A4000 16 GB | 1, 2, 4x | $0.76 |
| Paperspace | Quadro P5000 16 GB | 1, 2, 4x | $0.78 |
| Paperspace | Quadro RTX 5000 16 GB | 1, 2, 4x | $0.82 |
| Paperspace | Quadro P6000 24 GB | 1, 2, 4x | $1.10 |
| Paperspace | RTX A5000 24 GB | 1, 2, 4x | $1.38 |
| Paperspace | RTX A6000 48 GB | 1, 2, 4x | $1.89 |
| Paperspace | V100 32 GB | 1, 2, 4x | $2.30 |
| Paperspace | V100 16 GB | 1x | $2.30 |
| Paperspace | A100 40 GB | 1x | $3.09 |
| Paperspace | A100 80 GB | 1, 2, 4, 8x | $3.19 |
** Computed values. This was needed when single GPU instances were not available.4 5
Disclaimer
Please note that cloud providers are constantly updating their offering, therefore there could be updates to this.
External links
- Deep Learning GPU Benchmarks, Lambda Labs, Retrieved July 15, 2023
- Open LLM-Perf Leaderboard, Hugging Face, Retrieved July 15, 2023
- the-full-stack/website/docs/cloud-gpus, GitHub, Retrieved July 15, 2023
- The Ultimate Guide to Cloud GPU Providers, Paperspace, Retrieved July 15, 2023
- CloudOptimizer, CloudOptimizer, Retrieved July 15, 2023
Share on LinkedIn
Cem has been the principal analyst at AIMultiple since 2017. AIMultiple informs hundreds of thousands of businesses (as per similarWeb) including 55% of Fortune 500 every month.
Cem’s work has been cited by leading global publications including Business Insider, Forbes, Washington Post, global firms like Deloitte, HPE and NGOs like World Economic Forum and supranational organizations like European Commission. You can see more reputable companies and resources that referenced AIMultiple.
Throughout his career, Cem served as a tech consultant, tech buyer and tech entrepreneur. He advised enterprises on their technology decisions at McKinsey & Company and Altman Solon for more than a decade. He also published a McKinsey report on digitalization.
He led technology strategy and procurement of a telco while reporting to the CEO. He has also led commercial growth of deep tech company Hypatos that reached a 7 digit annual recurring revenue and a 9 digit valuation from 0 within 2 years. Cem’s work in Hypatos was covered by leading technology publications like TechCrunch like Business Insider.
Cem regularly speaks at international technology conferences. He graduated from Bogazici University as a computer engineer and holds an MBA from Columbia Business School.
Source link
