Setting up an in-house LLM platform: Best practices for optimal performance

Dos and don’ts of setting up an in-house LLM platform

Here are the dos and don'ts we figured out based on our experience while setting up a private AI cloud for one of our clients.

1. Hardware

This is the hard part i.e. investing in good reliable hardware. This is going to be expensive, but a one-time expense, convincing yourself and the company might be a little tough, but good LLMs need a lot of compute power to be useful in real time. Dos

• • Go for good GPUs and scalable options.

• • For small and medium applications, An H100 will be enough to handle most LLMs. You can use our recommendation chart below to see the compute power needed.

  To calculate your memory requirement, you can use the following formula:

Formula for Memory Requirement:	M=((P×4B)/(32/Q)) ×1.2
	Where M = Memory P = Parameters Q = Quantization

Model	Parameters	Precision (bits)	Approx. GPU Memory Requirement for 1 instance (GB)	Recommended Minimum GPU  (number of GPUs)	Memory (gb)	Number of GPU(s)
Llama 3.1	8B	32	38.4	H100	40	1
		16	19.2	A10	24	1
		8	9.6	T4	16	1
	70B	32	336	H100	80	5
		16	168	H100	80	2
		8	84	A10	4	4
		4	42	H100	80	1
	450B	32	1944	H100	80	25
		16	972	H100	80	13
		8	486	H100	80	7
		4	243	H100	80	4
Llama 2	13B	32	62.4	H100	80	1
		16	31.2	H100	40	1
		8	15.6	A10	24	1
		4	7.8	T4	16	1

Don'ts Don't try to set up the infrastructure with only CPUs. Although it may seem reasonable in the beginning, as soon as you start using the system, it will become evident that LLMs need compute power that is beyond the power of CPUs. Even if you are able to run it in the beginning, soon you will have scalability problems and then have to invest in the hardware again, doubling the cost. Recommended hardware: NVIDIA GPUs, TPU Pods

2. Architecture

Setting up a private AI cloud infrastructure requires an architecture that is scalable, secure, and capable of high performance. Dos Based on your requirements, you can use a Layered architecture, containerized architecture, or a hybrid of both.

• • Layered architecture is best for large structured systems, where the structure is defined and changes are not that frequent. Traditionally, enterprise companies follow this architecture as it is considered secure and compliance-friendly.

• • Containerized architecture, on the other hand, is more suitable for the rapid evolution of the system. Most Agile/DevOps teams follow containerized architecture, as containers are portable.

 

Note: Layered architecture separates components by functionality; containerized uses microservices packaged in containers.

Recommended architecture: Choose based on your use case, a hybrid of both would also be suitable if deciding is difficult. Don'ts

• • Please don't ignore the scalability needs. It might feel like you are ok with the current setup, but your architecture should be designed in a way that can handle scalability needs.

3. Setting up tools & frameworks

Now you have the hardware, and the architecture figured out, now is the time for selecting the model and the preferred framework. Do's

• • When selecting a model, ensure it is from an official trusted repo (e.g., an official hugging face repo).

• • When choosing frameworks for optimization and serving, ensure the optimized model format is compatible with the serving framework.

• • Choose optimization and serving frameworks compatible with the hardware (CPU or GPU) and have good community support.

  Recommended tools & frameworks

1. 1. AI frameworks: OpenAI, Nemo, Triton/vLLM, TensorRT-LLM, Hugging Face Transformers. Llama.cpp (hardware: CPU)

1. 1. Virtualization: VMware, Kubernetes, Docker.

1. 1. Storage: Azure Blob Storage.

1. 1. Security tools: HashiCorp Vault, Istio, Open Policy Agent (OPA).

  Don'ts

• • Don't ignore community support.

• • Choose tools and frameworks with tons of community support available online, as limited to no documentation will lead you spiraling in circles and wasting time and money.

 

4. Maintenance

Dos

• • Monitoring is an important aspect of private clouds. Since all infrastructure, models, tools, and frameworks are set up and owned by your company, monitoring it all is the only way you can keep it safe, secure, bias-free, and optimized.

  Recommended monitoring tools: Prometheus, Grafana. Don'ts

• • Don't ignore regular updates, as outdated versions might have vulnerabilities and not-so-satisfactory results as time passes.

 

Conclusion

Setting up an in-house LLM platform is not just about leveraging cutting-edge AI technology, it’s about aligning that technology with your business’s unique needs for data privacy, regulatory compliance, and cost efficiency. By carefully choosing the right hardware, designing a scalable and secure architecture, and selecting trusted tools and frameworks, organizations can create a platform that delivers high performance and complete control. By following the guidelines outlined, your business can build a future-ready AI platform tailored to its goals. Looking for expertise in generative AI? Reach out to us at marketing@confiz.com and talk to our experts.