OnPrem.LLM: A Privacy-Conscious Document Intelligence Toolkit

2025/05/12 by Arun S. Maiya, Maiya, Arun S. · 2 voices · 2 citations
Computer Science · #Big Data and Digital Economy #Computational Physics and Python Applications #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2505.07672

openalex publication_date 2025/05/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present OnPrem.LLM, a Python-based toolkit for applying large language models (LLMs) to sensitive, non-public data in offline or restricted environments. The system is designed for privacy-preserving use cases and provides prebuilt pipelines for document processing and storage, retrieval-augmented generation (RAG), information extraction, summarization, classification, and prompt/output processing with minimal configuration. OnPrem.LLM supports multiple LLM backends -- including llama.cpp, Ollama, vLLM, and Hugging Face Transformers -- with quantized model support, GPU acceleration, and seamless backend switching. Although designed for fully local execution, OnPrem.LLM also supports integration with a wide range of cloud LLM providers when permitted, enabling hybrid deployments that balance performance with data control. A no-code web interface extends accessibility to non-technical users.

Citations

• GPT4All: An Ecosystem of Open Source Compressed Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• Efficient Few-Shot Learning Without Prompts
• HuggingFace's Transformers: State-of-the-art Natural Language Processing

Cited by

• Reflective Personalization Optimization: A Post-hoc Rewriting Framework for Black-Box Large Language Models
• Generative AI for FFRDCs

Discussions

• OnPrem.LLM: A Privacy-Conscious Document Intelligence Toolkit [hn, 5 points, 0 comments]
• OnPrem.LLM: A Privacy-Conscious Document Intelligence Toolkit https://arxiv.org/abs/2505.07672 [bsky, 0 points, 0 comments]

Related

• PPMI: Privacy-Preserving LLM Interaction with Socratic Chain-of-Thought Reasoning and Homomorphically Encrypted Vector Databases
• Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devices
• Confidential Prompting: Privacy-preserving LLM Inference on Cloud
• LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMs
• RemoteRAG: A Privacy-Preserving LLM Cloud RAG Service