arxiv:2507.01853

Eka-Eval : A Comprehensive Evaluation Framework for Large Language Models in Indian Languages

Published on Jul 2, 2025

· Submitted by

Rajvee Sheth on Jul 7, 2025

#3 Paper of the day

Lingo Research Group

Upvote

Authors:

Samridhi Raj Sinha ,

Rajvee Sheth ,

Abstract

EKA-EVAL is a comprehensive evaluation framework for Large Language Models that includes diverse benchmarks, supports distributed inference, and is tailored for both global and Indic languages.

AI-generated summary

The rapid advancement of Large Language Models (LLMs) has intensified the need for evaluation frameworks that go beyond English centric benchmarks and address the requirements of linguistically diverse regions such as India. We present EKA-EVAL, a unified and production-ready evaluation framework that integrates over 35 benchmarks, including 10 Indic-specific datasets, spanning categories like reasoning, mathematics, tool use, long-context understanding, and reading comprehension. Compared to existing Indian language evaluation tools, EKA-EVAL offers broader benchmark coverage, with built-in support for distributed inference, quantization, and multi-GPU usage. Our systematic comparison positions EKA-EVAL as the first end-to-end, extensible evaluation suite tailored for both global and Indic LLMs, significantly lowering the barrier to multilingual benchmarking. The framework is open-source and publicly available at https://github.com/lingo-iitgn/ eka-eval and a part of ongoing EKA initiative (https://eka.soket.ai), which aims to scale up to over 100 benchmarks and establish a robust, multilingual evaluation ecosystem for LLMs.

View arXiv page View PDF Project page GitHub 30 Add to collection

Community

RajveeSheth

Paper author Paper submitter Jul 7, 2025

•

edited 20 days ago

Abstract
The rapid evolution of Large Language Models has underscored the need for evaluation frameworks that are globally applicable, flexible, and modular, and that support a wide range of tasks, model types, and linguistic settings. We introduce EKA-EVAL, a unified, end-to-end framework that combines a zero-code web interface and an interactive CLI to ensure broad accessibility. It integrates 55+ diverse benchmarks across nine evaluation categories, supports local and proprietary models, and provides 11 core capabilities through a modular, plug-and-play architecture. Designed for scalable, multilingual evaluation with support for low-resource multilingual languages, EKAEVAL is, to the best of our knowledge, the first suite to offer comprehensive coverage in a single platform. Comparisons against five existing baselines indicate improvements of at least 2x better on key usability measures, with the highest user satisfaction, faster setup times, and consistent benchmark reproducibility.