Tasks/LLM Benchmarks
LLM Benchmarks
বৃহৎ ভাষা মডেল মূল্যায়নEvaluating large language models on Bangla knowledge, reasoning, and generation.
8 datasets·92 papersTop listed · Claude-3.5-Sonnet — 68.53 Accuracy (answer-only) (2025)
| Dataset | Size | License | Year ▾ | Source paper | Link |
|---|---|---|---|---|---|
BanglaSocialBench Sociopragmatic LLM benchmark over Bangla address terms, kinship reasoning, and social customs. | 1,719 instances | CC BY 4.0 | 2026 | BanglaSocialBench (ACL SRW 2026) | ✓ verified 2026-08-23 |
B-REASO Bengali multiple-choice evaluation suite over 50 subjects at four difficulty levels, plus the harder B-REASO HEAVY subset. | 13,497 questions | MIT | 2025 | B-REASO (Findings EMNLP 2025) | ✓ verified 2026-07-19 |
BnMMLU Bengali massive multitask language understanding benchmark spanning 23 domains. | 134,375 question-option pairs | CC BY-SA 4.0 | 2025 | BnMMLU (2025) | ✓ verified 2026-07-19 |
BanglaMATH Bangla grade 6-8 math word problems for testing LLM mathematical reasoning. | ~1,700 problems | Research only | 2025 | BanglaMATH (MathNLP 2025) | ✓ verified 2026-07-19 |
SOMADHAN Bengali math word problems with step-by-step solutions adapted from the GSM8K train and test sets. | 8,792 problems | CC BY 4.0 | 2025 | SOMADHAN (2025) | ✓ verified 2026-07-26 |
BEnQA Parallel Bangla–English science exam questions for LLM evaluation. | 5,161 questions | CC BY-NC-SA 4.0 | 2024 | BEnQA (2024) | ✓ verified 2026-07-19 |
Global-MMLU (bn) Culturally adapted MMLU; Bengali split. | 14,327 questions | Apache 2.0 | 2024 | Global-MMLU (2024) | ✓ verified 2026-07-19 |
MEGAVERSE (bn subset) Multilingual LLM evaluation suite including Bangla tasks. | 22 datasets | Mixed | 2024 | MEGAVERSE (NAACL 2024) | ✓ verified 2026-08-23 |
Results listed for B-REASO · metric: Accuracy (answer-only)
No rows curated for BEnQA (bn) yet — contribute via PR.
Scores are community-reported; see linked papers for official numbers. Report a correction.
2026BaFCo: A Document Understanding Benchmark for Complex Bangla Form ComprehensionAzad et al.arXiv2026BanglaLlama: LLaMA for Bangla LanguageZehady et al.LoResLM2026BanglaLorica: Design and Evaluation of a Robust Watermarking Algorithm for Large Language Models in Bangla Text GenerationTariqul et al.arXiv2026BanglaMemeEvidence: A Multimodal Benchmark Dataset for Explanatory Evidence Detection in Bengali MemesFaria et al.arXiv2026BanglaSocialBench: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social InteractionSijan et al.ACL2026Behind the Laughter: Uncovering Gender Bias in Code-Mixed Bangla MemesFerdusi et al.Sixth Workshop on Language Technology for Equality, Diversity, Inclusion2026Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIFSami et al.arXiv2026BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on BengaliAdib et al.arXiv2026BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali ContextsNoshin et al.arXiv2026Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky RiddlesSayeedi et al.*SEM2026Cross-Lingual Probing and Community-Grounded Analysis of Gender Bias in Low-Resource BengaliReaj et al.arXiv2026CUET-823 at SemEval-2026 Task 9: LoRA-Based Instruction Fine-Tuning of LLMs vs. Transformer Models for Bengali Polarization DetectionMallik et al.SemEval2026GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPORoy Dipta et al.Findings ACL2026GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language ModelsPaul et al.arXiv2026How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and BenchmarkingAhmed et al.arXiv2026Mina: A Multilingual LLM-Powered Legal Assistant Agent for Empowering Access to Justice in BangladeshWasi et al.Findings ACL2026MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social MediaPramanik et al.arXiv2026Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla LanguageNandi et al.arXiv2026When a Name Is Not a Name: A Benchmark Dataset and Distilled Reasoning for Culturally Entangled Bangla Homographs in Low-Resource LLMsShuvo et al.arXiv2025Assessing Gender Bias of Pretrained Bangla Language Models in STEM and SHAPE FieldsArnob et al.GeBNLP2025Auditing Political Bias in Text Generation by GPT-4 using Sociocultural and Demographic Personas: Case of Bengali Ethnolinguistic CommunitiesDas et al.Workshop on Benchmarks, Harmonization, Annotation, and Standardization for Human-Centric AI in Indian Languages2025B-REASO: A Multi-Level Multi-Faceted Bengali Evaluation Suite for Foundation ModelsHosain and Morol · 13,497 MCQs over 50 subjects at four difficulty levels; includes the harder B-REASO HEAVY subset.Findings EMNLP2025BanglaForge: LLM Collaboration with Self-Refinement for Bangla Code GenerationDihan et al.arXiv2025BanglaMATH : A Bangla benchmark dataset for testing LLM mathematical reasoning at grades 6, 7, and 8Prama et al.Workshop on Mathematical Natural Language Processing2025BanStereoSet: A Dataset to Measure Stereotypical Social Biases in LLMs for BanglaKamruzzaman et al.Findings ACL2025BengaliFig: A Low-Resource Challenge for Figurative and Culturally Grounded Reasoning in BengaliAl SefatWorkshop on Multimodal Models for Low-Resource Contexts and Social Impact2025BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and CultureRidoy et al.arXiv2025BenNumEval: A Benchmark to Assess LLMs' Numerical Reasoning Capabilities in BengaliAhmed et al.Findings ACL2025BLUCK: A Benchmark Dataset for Bengali Linguistic Understanding and Cultural KnowledgeKabir et al.BLP @ IJCNLP-AACL2025BnMMLU: Measuring Massive Multitask Language Understanding in BengaliJoy and Shatabda · 134,375 question-option pairs across 23 domains.arXiv2025BRACU\_CL at BLP-2025 Task 2: CodeMist: A Transformer-Based Framework for Bangla Instruction-to-Code GenerationJuboraj et al.BLP @ IJCNLP-AACL2025Code\_Gen at BLP-2025 Task 2: BanglaCode: A Cross-lingual Benchmark for Code Generation with Translation and Assertion StrategiesAgarwala et al.BLP @ IJCNLP-AACL2025CodeAnubad at BLP-2025 Task 2: Efficient Bangla-to-Python Code Generation via Iterative LoRA Fine-Tuning of Gemma-2RoyBLP @ IJCNLP-AACL2025destroR: A Benchmark and Adversarial-Training Defense for Bangla Transfer Models under Meaning-Preserving AttacksAhmed et al.arXiv2025Empowering Low-Resource Languages: TraSe Architecture for Enhanced Retrieval-Augmented Generation in BanglaIpa et al.Workshop on Language Models for Underserved Communities2025End-to-End Bangla AI for Solving Math Olympiad Problem Benchmark: Leveraging Large Language Model Using Integrated ApproachTabib et al.arXiv2025Evaluating Credibility and Political Bias in LLMs for News Outlets in BangladeshPrama et al.ACL2025Evaluating LLMs’ Multilingual Capabilities for Bengali: Benchmark Creation and Performance AnalysisBhowmik et al. · Eight translated datasets evaluated across ten open-source LLMs.arXiv2025Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPEPatwary et al.arXiv2025From Facts to Folklore: Evaluating Large Language Models on Bengali Cultural KnowledgeChowdhury et al.IJCNLP2025Generation, Analysis, and Detection of LLM Manipulated Bangla NewsAkther et al.IEEE Access2025Intrinsic Linguistic Bias in Formal vs. Informal Bengali Pragmatics with Progressive Context InflationHosain et al.IJCNLP2025JU\_NLP at BLP-2025 Task 2: Leveraging Zero-Shot Prompting for Bangla Natural Language to Python Code GenerationPal et al.BLP @ IJCNLP-AACL2025Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought ReasoningPaul et al.arXiv2025Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM IntegrationGharami et al.arXiv2025Musafir at BLP\_2025 Task 2: Generating Python Code from Bangla Prompts using a Multi Model Cascade and Unit Test ValidationHasan et al.BLP @ IJCNLP-AACL2025NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code GenerationSaadi et al.arXiv2025NALA\_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bengali Instruction to Python Code GenerationSaadi et al.BLP @ IJCNLP-AACL2025NSU\_PiedPiper at BLP-2025 Task 2: A Chain-of-Thought with Iterative Debugging Approach for Code Generation with Bangla InstructionFahmid et al.BLP @ IJCNLP-AACL2025Overview of BLP-2025 Task 2: Code Generation in BanglaRaihan et al.BLP @ IJCNLP-AACL2025PyBangla at BLP-2025 Task 2: Enhancing Bangla-to-Python Code Generation with Iterative Self-Correction and Multilingual AgentsIslam et al.BLP @ IJCNLP-AACL2025PyBhasha at BLP-2025 Task 2: Effectiveness of Semantic-Aware Translation and Ensembling in Bangla Code GenerationDewan et al.BLP @ IJCNLP-AACL2025Read Between the Lines: A Benchmark for Uncovering Political Bias in Bangla News ArticlesLia et al.BLP @ IJCNLP-AACL2025Retriv at BLP-2025 Task 2: Test-Driven Feedback-Guided Framework for Bangla-to-Python Code GenerationAsib et al.BLP @ IJCNLP-AACL2025Reveal-Bangla: A Dataset for Cross-Lingual Multi-Step Reasoning EvaluationIslam et al.BLP @ IJCNLP-AACL2025Robustness of LLMs to Transliteration Perturbations in BanglaHaider et al.BLP @ IJCNLP-AACL2025Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious BiasSadhu et al.LoResLM2025Structured Reasoning with Tree-of-Thoughts for Bengali Math Word ProblemsMahmood et al.arXiv2025TeamB2B at BLP-2025 Task 2: BanglaForge: LLM Collaboration with Self-Refinement for Bangla Code GenerationDihan et al.BLP @ IJCNLP-AACL2025TigerCoder: A Novel Suite of LLMs for Code Generation in BanglaRaihan et al.arXiv2025TigerLLM - A Family of Bangla Large Language ModelsRaihan et al.ACL2025TituLLMs: A Family of Bangla LLMs with Comprehensive BenchmarkingNahin et al. · Bangla-adapted Llama models.arXiv2025Troopers at BLP-2025 Task 2: Reward-Selective Fine-Tuning based Code Generation Approach for Bangla PromptsFarazi et al.BLP @ IJCNLP-AACL2025Women, Infamous, and Exotic Beings: A Comparative Study of Honorific Usages in Wikipedia and LLMs for Bengali and HindiMukherjee et al.EMNLP2025WoNBias: A Dataset for Classifying Bias \& Prejudice Against Women in Bengali TextAupi et al.GeBNLP2024An Empirical Study of Gendered Stereotypes in Emotional Attributes for Bangla in Multilingual Large Language ModelsSadhu et al.GeBNLP2024An Empirical Study on the Characteristics of Bias upon Context Length Variation for BanglaSadhu et al.Findings ACL2024BenLLM-Eval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLPKabir et al. · Zero-shot LLMs vs fine-tuned baselines.LREC-COLING2024BEnQA: A Question Answering Benchmark for Bengali and EnglishShafayat et al. · Parallel science exam questions; large en–bn gap.Findings ACL2024BN-AuthProf: Benchmarking Machine Learning for Bangla Author Profiling on Social Media TextsTasnim et al.arXiv2024Exploring Bengali Religious Dialect Biases in Large Language Models with Evaluation PerspectivesWasi et al.arXiv2024Harnessing Large Language Models Over Transformer Models for Detecting Bengali Depressive Social Media Text: A Comprehensive StudyChowdhury et al.arXiv2024Improving Bengali and Hindi Large Language ModelsShahriar et al.LREC-COLING2024Medical nearest-word embedding technique implemented using an unsupervised machine learning approach for Bengali languageMandal et al.International Journal on Smart Sensing and Intelligent Systems2024Mitigating Extrinsic Gender Bias for Bangla Classification TasksJoy et al.arXiv2024Uddessho: An Extensive Benchmark Dataset for Multimodal Author Intent Classification in Low-Resource Bangla LanguageFaria et al.arXiv2023Assessing Political Inclination of Bangla Language ModelsThapa et al.BLP @ EMNLP2023Bengali Intent Classification with Generative Adversarial BERTHasan et al.arXiv2023BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLPKabir et al.arXiv2023ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource LanguagesGhosh et al.arXiv2023Crosslingual Retrieval Augmented In-context Learning for BanglaLi et al.BLP @ EMNLP2023Next Words Prediction and Sentence Completion in Bangla Language Using GRU-Based RNN on N-Gram Language ModelHoque et al.Journal of Data Analysis and Information Processing2023Toward Cultural Bias Evaluation Datasets: The Case of Bengali Gender, Religious, and National IdentityDas et al.First Workshop on Cross-Cultural Considerations in NLP2022Bangla text generation system by incorporating attention in sequence-to-sequence modelBanik et al.World Journal of Advanced Research and Reviews2022Bangla-BERT: Transformer-Based Efficient Model for Transfer Learning and Language UnderstandingKowsher et al.IEEE Access2022BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in BanglaBhattacharjee et al. · Introduces BanglaBERT + the BLUB benchmark.Findings NAACL2021BnVec: Towards the Development of Word Embedding for Bangla Language ProcessingKowsher et al.International Journal of Engineering & Technology2021Context-driven Bengali Text Generation using Conditional Language ModelKibria et al.Statistics Optimization & Information Computing2021Distributed Deep Learning in Open CollaborationsDiskin et al. · Collaboratively trained Bangla ALBERT.NeurIPS2019An investigative design of optimum stochastic language model for bangla autocompleteEyamin et al.Indonesian Journal of Electrical Engineering and Computer Science2018An Exploratory Approach to Find a Novel Metric Based Optimum Language Model for Automatic Bangla Word PredictionHabib et al.International Journal of Intelligent Systems and Applications2015Automated Word Prediction in Bangla Language Using Stochastic Language ModelsHaque et al.International Journal in Foundations of Computer Science & Technology
BASEcsebuetnlp/banglabert
ELECTRA · 110M
Pretrained Bangla ELECTRA discriminator evaluated on the BLUB benchmark.
✓ verified 2026-06-28
csebuetnlp/banglabert_large
ELECTRA · 335M
✓ verified 2026-06-28
csebuetnlp/banglishbert
ELECTRA · 110M
Code-mixed Bangla-English encoder.
✓ verified 2026-05-11
sagorsarker/bangla-bert-base
BERT · 165M
✓ verified 2026-05-11
neuropark/sahajBERT
ALBERT · 18M
Collaboratively trained.
✓ verified 2026-03-19
google/muril-base-cased
BERT · 236M
17 Indian languages including Bangla.
✓ verified 2026-03-19
hishab/titulm-llama-3.2-3b-v2.0
Llama 3.2 · 3B
Continually pretrained Bangla LLM.
✓ verified 2026-06-28
hishab/titulm-llama-3.2-1b-v2.0
Llama 3.2 · 1B
Continually pretrained on 37B Bangla tokens with an extended tokenizer.
✓ verified 2026-07-26