[ad_1]
As utilizing “big data” is more and more relevant for problem-solving across every industry, data repositories of homelab and data-lake scale alike require more parallelized computing power to extract, transform, load, and analyze data than ever before. While creating my own homelab, the decision to create my parallelized setups over virtual machines or natively on hardware left me stumped, and I struggled to find performance comparisons. In this article, we’ll explore some of the pros and cons of each setup, as well as a side-by-side performance and benchmarks of each methodology both virtual and native.
Many of parallelized compute clusters include multiple nodes, or computers designated to process tasks distributed over them in a cluster. Managing such nodes can be a major headache, hence why Data Engineering is so lucrative compared to their analytical counterparts. Typically, companies will manage entire fleets of clusters, which would make it almost impossible to give individual attention to individual nodes, and instead “high availability” setups with tools such as Proxmox, Kubernetes, and Docker Swarm are requirements for the modern enterprise. You’ve likely already interacted with these clusters and not realized this week, even — the chicken sandwich I had for lunch from Chick-fil-A is famously fulfilled via an edge-computing Kubernetes cluster with their point-of-sale system.
There are many benefits to computing in virtualized machines, including:
- Entire operating systems can be rapidly deployed from corporate servers to the field nearly instantaneously
- Images can be backed up in real-time
- Deployments can be containerized to limit scope and increase security
- In the event of hardware failures, systems can be migrated with minimal downtime
These are not new concepts by any means, but with a growing need for data analysis at every level of organizations, the way…
Source link
