Proxmox VE as the basis for locally hosted AI services

As a leading provider of software solutions for digitization in companies, otris software AG relies on Proxmox VE to offer privacy-friendly AI services. The company operates locally hosted Large Language Models (LLMs) in virtual machines using NVIDIA GPUs. Thanks to the robustness and flexibility of Proxmox VE, operation is guaranteed even if a server fails.

When otris software AG planned to integrate AI functions into its products to optimize document management and exploit the full potential of the available data, one of the biggest challenges was to operate Large Language Models (LLMs) securely in a private cloud managed by a subsidiary, otris systems GmbH, in three German data centers.

According to Dr. Christoph Niemann, co-founder and CEO of otris software AG, key requirements had to be met to implement a powerful and privacy-friendly solution: Replication between two data centers should ensure load balancing and reliability, the use of virtual machines should provide a flexible environment for the LLMs, and the direct assignment of NVIDIA H100 GPUs to virtual machines via PCIe passthrough should enable dynamic GPU utilization.

Flexibility and reliability with Proxmox VE

Proxmox VE proved to be the ideal solution for these requirements, as it enables the reliable and efficient management of virtual instances. The ease of use and robust architecture of Proxmox VE allows for flexible use and scaling. ZFS enables fast and efficient storage replication.

In practical use, the solution proved to be extremely stable and reliable, and the installation and operation of Proxmox VE on Supermicro hardware went smoothly.

Failover with two identically configured systems

Two identically configured Supermicro 4125GS-TNRT systems were used for the implementation, each equipped with two AMD EPYC 9654 processors with 96 cores, 1.5 TB DDR5 RAM and a capacity of up to eight GPUs per server. The network structure is based on a 10GbE connection for external connections and a separate 100GbE network for internal communication. The GPUs are forwarded directly to the VMs via PCIe passthrough.

An unexpected defect in the mainboard of one of the two servers revealed the high performance of the solution: thanks to the flexible infrastructure, the virtual machine was quickly transferred to the second server, the available GPU was integrated, and operations were resumed within a very short time.

This area illustrates the strong innovative power of Proxmox, since Proxmox VE is an NVIDIA vGPU supported hypervisor, and from version 8.4 even allows the live migration of “mediated devices” such as NVIDIA vGPUs, increasing reliability even further.

Maximum scalability and optimal use of resources

The long-term benefits of implementing Proxmox VE are manifold. The virtualization solution offers maximum scalability, as the existing server design can be expanded with additional GPUs at any time. In addition, Proxmox VE enables optimal resource utilization through dynamic distribution of CPU cores, RAM, and GPUs, ensuring particularly precise and efficient use of the hardware.

In addition, the open-source solution from Proxmox allows considerable cost savings, as licensing costs are eliminated and investments can flow more specifically into high-performance hardware. The environment remains flexible and future-proof, as it can be easily expanded and adapted to new requirements.

Another advantage for otris software AG is the stability of Proxmox VE: the continuous updates ensure secure operation with minimal downtime.

Future-proof platform for AI based on Proxmox VE

The combination of Proxmox VE and powerful hardware offers otris software AG a reliable, cost-efficient, and future-proof platform for AI-supported processes. Thanks to Proxmox VE’s high scalability, flexibility and stability, the company is ideally equipped to continue using innovative technologies efficiently in the future.

“It was important to us that customers of our otris legal suite have a choice: To use the common AI models at the well-known hosters or to use a local AI so that customer data does not leave the otris systems,” says Dr. Christoph Niemann. “For the latter, we take advantage of our own LLM on Proxmox VE with NVIDIA GPU cards in our own environment, which is maintained completely by our colleagues at otris systems GmbH.”

In the future, the company is going to continue to rely on the versatility and innovative power of Proxmox VE. There are already plans to switch from storage replication with ZFS to the distributed storage system Ceph to improve real-time replication between data centers, which will significantly increase redundancy and operational reliability.

Dr. Christoph Niemann
Co-founder and CEO

About otris software AG
As a leading provider of software solutions for digitization, otris software AG supports companies on their path to digitalization. Its software solutions are based on its own platform for document-centric business processes which makes complex issues transparent and enables well-founded decisions. Its specialist solutions support decision-makers in efficiently fulfilling their management responsibilities in the areas of legal & administration, compliance, and data protection. They are characterized by a high level of usability and integrate state-of-the-art technologies such as artificial intelligence (AI) to meet the highest demands.

About Proxmox partner: otris systems GmbH
Since 2003 otris systems GmbH has been offering customized solutions in the field of open-source software. The company focuses on needs-based consulting, transparent implementation, and the sustainable and secure operation of IT infrastructure and applications. By operating its own hosting environments, otris systems is able to implement individual cloud solutions according to customer specifications. At the same time, operation of on-premises infrastructure is equally one of the core competencies of otris systems GmbH. Complex hybrid environments that combine the advantages of both worlds are therefore just as much a part of the day-to-day business of otris systems as the support of individual systems. More at: https://otris.systems