Co-authors: Wei Wen Devariah Christihapsari Isaac Tan

Abstract

We tested two large language models (LLMs), GPT-4o-mini and SEA-LIONv3, on their handling of Indonesian-specific questions. Through human expert evaluation of 50 questions across four categories, we found that SEA-LIONv3 excelled at understanding local details and nuances, while both models performed comparably on general language tasks. This difference highlights the potential of localized LLMs like SEA-LIONv3 to enhance accuracy and cultural sensitivity in Southeast Asian applications.

Introduction

The increasing use of large language models (LLMs) across Southeast Asia raises essential questions about their ability to effectively navigate the region’s unique linguistic and cultural landscape. Successful LLM deployment in Southeast Asia requires generating contextually relevant and culturally appropriate text. Given Southeast Asia’s rich mosaic of languages, traditions, and social norms, this need for cultural sensitivity is crucial.

We compared the performance of two LLMs: GPT-4o-mini, a generalized model, and SEA-LIONv3, a model fine-tuned explicitly for Southeast Asian contexts. By evaluating their responses to a range of Indonesian-specific questions, we aim to highlight the binding role of region-specific fine-tuning on LLM performance in Southeast Asia. Ultimately, this research helps guide the development of responsible and effective regional AI applications in the region.

Model Specifications

• GPT-4o-mini: A generalized model supporting multiple languages beyond Southeast Asia
• SEA-LIONv3: A region-focused model supporting 13 languages, including English, Chinese, Vietnamese, Indonesian, Thai, Filipino, Tamil, Malay, Khmer, Lao, Burmese, Javanese, and Sundanese

Evaluation Design

We developed 50 questions across four categories:

• Language-based: To assess the models’ understanding of Indonesian grammar, vocabulary, and linguistic nuances.
• Domain-based: To evaluate the models’ knowledge of specific fields relevant to Indonesian history, culture, economy, and environment.
• Geographical-based: To assess the understanding of local geography, cultural traditions, and norms.
• Combined: To test integrated knowledge across multiple categories.

Each question was presented in two variations: English and Indonesian. The data is available on HuggingFace. We acknowledge that 50 questions may not be comprehensive enough to evaluate the models across all possible scenarios. However, it provides an initial benchmark for assessing the models’ performance.

Evaluator Background

Devariah Christihapsari, a Master’s student in Public Administration at the University of Malaya, evaluated the LLMs’ responses. Having spent her formative years immersed in the diverse cultures of Asia, including Vietnam and China, Devariah developed a deep appreciation for the nuances of language and tradition. Her passion for history and geography further enriched her understanding of Indonesia’s intricate cultural landscape. This personal connection enabled her to approach the evaluation with a keen eye for detail, ensuring that the LLMs’ responses were assessed for their factual accuracy and sensitivity to the subtle complexities of Indonesian social norms and cultural values.

Evaluation Process

Both models were prompted with each question on the eval platform hosted by SUPA. Responses were then evaluated by a Master’s student in Public Administration, a native Indonesian speaker. The evaluation criteria were as follows:

• Relevance: How well the response addresses the prompt.
• Coherence: The logical flow and organization of the response.
• Factuality: The accuracy of the information provided.
• Creativity: The originality and inventiveness of the response.
• Tone and Style: The appropriateness of the language and style for the given context.

The evaluator provided detailed feedback on each response, and the results were compiled and analyzed to compare the performance of the two models. The complete evaluation results and the evaluator’s feedback are available here for full transparency.

Results

The results show that SEA-LIONv3 outperformed GPT-4o-mini in all categories except language-based questions, where both models demonstrated comparable performance. SEA-LIONv3 particularly excelled in the domain-based category. According to our evaluator, while GPT-4o-mini exhibited stronger grammatical accuracy, SEA-LIONv3 demonstrated greater contextual understanding.

Table 1: Model Performance on Indonesian Context-Specific Tasks

Figure 1: Stacked bar chat visualizing the performance of both models across categories.

Analysis by Category

1. Language-based: Both models demonstrated a strong grasp of Indonesian grammar, vocabulary, and sentence structure, as evidenced by their comparable scores in this category. However, SEA-LIONv3 generally exhibited a more nuanced grasp of the Indonesian language, particularly in its ability to:

• Capture regional variations and slang: SEA-LIONv3 was often better at recognizing and explaining slang terms and regional variations in language use.
• Explain idioms accurately: SEA-LIONv3 consistently provided more accurate and nuanced explanations of Indonesian idioms.
• Understand the cultural context of language: SEA-LIONv3 demonstrated a greater awareness of the cultural context in which language is used, particularly concerning politeness and formality.

2. Domain-based: SEA-LIONv3 significantly outperformed GPT-4o-mini in this category. SEA-LIONv3 possesses a broader and deeper understanding of Indonesian history, culture, economy, and environment. This advantage likely stems from SEA-LIONv3’s region-specific fine-tuning, which has equipped it with more comprehensive and nuanced knowledge relevant to Indonesia. Specifically, SEA-LIONv3 excelled in:

• E-commerce and Digital Payments: SEA-LIONv3 was generally more accurate in identifying popular platforms and methods.
• Cultural Practices and Traditions: SEA-LIONv3 provided more detailed and accurate descriptions of various cultural aspects, including traditional arts, ceremonies, and social norms.
• Sub-Ethnicities: SEA-LIONv3 demonstrated a deeper understanding of the cultural differences and unique characteristics of various Indonesian sub-ethnicities.
• Geography and Tourist Destinations: SEA-LIONv3 was more accurate in identifying popular tourist spots and describing Indonesian geographical features

3. Geographical and Cultural-based: SEA-LIONv3 again showed a clear advantage in this category, demonstrating greater accuracy in answering questions related to Indonesian geography, cultural traditions, and social norms. This suggests that SEA-LIONv3’s training data included richer information about Indonesia’s diverse cultural landscape.

• Administrative Divisions and Geography: SEA-LIONv3 provided more detailed and deeper explanations of Indonesia’s provinces, islands, and geographical features.
• Cities and Locations: SEA-LIONv3 was more accurate in identifying major Indonesian cities and their geographical context.
• Climate Zones: SEA-LIONv3 offered more comprehensive descriptions of Indonesia’s diverse climate zones and their characteristics.
• Unique Geographical Features: SEA-LIONv3 identified unique geographical features and explained their cultural significance.
• Transportation Options: SEA-LIONv3 provided more detailed and inclusive information about various transportation methods used in Indonesia.

4. Combined: SEA-LIONv3 excelled in tasks requiring nuanced understanding and detailed elaboration, particularly those involving cross-domain knowledge and complex cultural, historical, and societal aspects of Indonesia. It demonstrated:

• Enhanced Contextual Understanding: SEA-LIONv3 incorporates cultural subtleties, such as vivid descriptions of Indonesian sub-ethnic traditions, hierarchical structures, and culinary influences.
• Improved Relevance and Clarity: By balancing factual accuracy with cultural context, the model delivers more relevant responses tailored to the audience’s needs. For instance, its elaboration on art, festivals, and social dynamics demonstrates a fine grasp of the topics.
• Versatility Across Topics: Whether summarizing travel plans, reporting natural disasters, or exploring societal values, SEA-LIONv3 adapts seamlessly, showcasing its ability to address diverse themes precisely.
• Superior Sensory Engagement: Particularly in categories like culinary traditions, SEA-LIONv3 uses suggestive language to create an immersive experience, connecting readers emotionally to the content

Table 2: Examples of Evaluation Prompts and Model Responses
For better visualization, check it out on Huggingface

Prompt	GPT-4o-mini (Model A)	SEA-LIONv3 (Model B)	Preferred Model
What are the most pressing environmental concerns in Indonesia?	Correct. 1. Deforestation: Indonesia has one of the highest rates of deforestation in the world, driven primarily by agriculture, logging, and land conversion for palm oil plantations. This loss of forest contributes to biodiversity loss, the displacement of indigenous communities, and increases carbon emissions. 2. Biodiversity Loss: The rich biodiversity of Indonesia is threatened by habitat destruction, pollution, and climate change. Many species, including orangutans, tigers, and various endemic plants, face extinction due to the rapid loss of their natural habitats. 3. Climate Change: Indonesia is one of the countries most vulnerable to climate change impacts, including rising sea levels, increased flooding, and changes in weather patterns that affect agriculture. The country’s extensive coastline also makes it susceptible to natural disasters exacerbated by climate change. 4. Air Pollution: Urban air quality is a significant concern, particularly in cities like Jakarta. Major sources include vehicle emissions, industrial discharges, and burning of waste. Additionally, seasonal haze caused by land and forest fires contributes to air quality issues in the region. 5. Water Pollution: Rivers and bodies of water in Indonesia face severe pollution from industrial discharges, agricultural runoff, and untreated sewage. This pollution threatens both public health and aquatic ecosystems. 6. Waste Management: Indonesia struggles with waste management, particularly plastic waste. The country is one of the largest contributors to ocean plastic pollution, with inadequate recycling systems and waste management practices leading to significant environmental impacts. 7. Land Degradation and Soil Erosion: Unsustainable agricultural practices and deforestation have led to land degradation, soil erosion, and loss of arable land. This affects food security and rural livelihoods. 8. Coral Reef Degradation: Indonesia has some of the world’s most biodiverse coral reefs, but they are threatened by overfishing, destructive fishing practices, coastal development, and climate change, including rising ocean temperatures and acidification.	Correct & nuanced. 1. Deforestation and Forest Degradation:Driven by palm oil plantations, logging, and mining, deforestation rates remain alarmingly high. This leads to: Biodiversity Loss:Indonesia is a global biodiversity hotspot, and habitat destruction threatens countless species, including orangutans, tigers, and endemic flora. Climate Change:Forests act as carbon sinks; deforestation releases massive amounts of CO2, contributing significantly to global warming. Soil Erosion and Landslides:Loss of forest cover increases vulnerability to soil erosion, landslides, and flooding. 2. Air Pollution: Major cities like Jakarta suffer from severe air pollution due to: Vehicle Emissions: A rapidly growing number of vehicles, coupled with inadequate public transport, contribute heavily. Industrial Activities:Factories and power plants release pollutants into the air. Forest Fires:Annual slash-and-burn practices for agriculture and land clearing generate massive smoke plumes, impacting air quality regionally. 3. Plastic Pollution:Indonesia is one of the world’s largest plastic polluters, with plastic waste clogging waterways, harming marine life, and contaminating beaches and ecosystems. Inadequate waste management systems exacerbate this issue. 4. Marine Degradation:Overfishing, destructive fishing practices (like dynamite fishing), and pollution threaten Indonesia’s rich marine biodiversity and vital fisheries, impacting livelihoods and food security. Coral reefs, crucial for coastal protection and tourism, are also suffering from bleaching and damage. 5. Water Resources Management: Growing populations and industrial demands strain freshwater resources. Pollution from agriculture, industry, and untreated sewage contaminates rivers and groundwater, affecting both human health and ecosystems.	Model B wins. While Model A is informative, it reads more like a list of separate issues. Model B, on the other hand, connects the dots and offers a more nuanced and comprehensive understanding of Indonesia’s environmental landscape. Model B included solutions, interconnectedness, and specificity of the problems.
What are some of the major sub-ethnicities within Indonesia, and how do their cultures differ?	Correct. 1. Javanese (Java Island): Known for their politeness, intricate arts like batik and wayang puppetry, and strong traditions of courtly etiquette. Islam plays a central role, often interwoven with ancient animistic beliefs. They value hierarchy and respect for elders. 2. Sundanese (West Java): Similar to Javanese in politeness but with a more direct communication style. Famous for their gamelan music, traditional dances like Jaipongan, and unique culinary traditions like nasi timbel (rice wrapped in banana leaves). Predominantly Muslim, they also retain elements of indigenous beliefs. 3. Batak (North Sumatra): Strong animistic roots with ancestor veneration playing a significant role. Known for their intricate wood carvings, unique traditional houses (honai), and vibrant festivals like the “Batu Gonggong” ceremony. Christianity is prevalent, alongside remnants of indigenous beliefs. Language: Batak dialects (Toba, Karo, Simalungun, etc.) 4. Minangkabau (West Sumatra): Matrilineal society where lineage and inheritance trace through the female line. Renowned for their “Rumah Gadang” (large communal houses), intricate weaving traditions, and strong coffee culture. Primarily Muslim with influences from ancient adat (customary law). 5. Balinese (Bali Island): Highly spiritual and artistic, deeply influenced by Hinduism (though with unique local interpretations). Celebrated for their elaborate temple complexes (pura), vibrant dances like Legong and Barong, and offerings (canang sari) central to daily life. 6. Dayak (Kalimantan Island): Diverse group with numerous tribes, each possessing unique customs and longhouse traditions. Known for their skilled hunting and farming practices, intricate tattoos, and spiritual connection to nature. Animistic beliefs and ancestor worship are prominent. 7. Papuan (Papua & West Papua): Indigenous peoples with rich traditions of tribal customs, body painting, and diverse musical styles. Strong connection to ancestral lands and nature. Religions range from indigenous beliefs to Christianity and Islam.	Correct and more nuanced answer. 1. Javanese: Location: Primarily on the island of Java. Culture: The Javanese culture is known for its rich traditions in arts, including batik (traditional textile), gamelan music, and wayang kulit (shadow puppetry). Javanese society values hierarchy and politeness (known as “santun”), and their philosophy often reflects a blend of animism, Hinduism, Buddhism, and Islam. 2. Sundanese: Location: West Java. Culture: Sundanese culture is characterized by its music, such as angklung (a bamboo musical instrument), and dance. The Sundanese often have a closer connection to nature and agriculture, with rice farming being central to their lifestyle. They are also known for their distinct dialect and culinary traditions. 3. Balinese: Location: Bali. Culture: Balinese culture is heavily influenced by Hinduism, evident in their ornate temples, elaborate ceremonies, and traditional dances like the Legong and Barong. Balinese art, particularly in painting and sculpture, is renowned for its detail and spirituality. Their rituals often involve community participation and a deep respect for nature. 4. Batak: Location: North Sumatra. Culture: The Batak people consist of several subgroups, such as the Toba, Karo, and Simalungun, each with its own customs and languages. Batak culture includes distinct traditional music and dance, unique wooden architecture, and an elaborate oral storytelling tradition. The Toba Batak, for example, are known for their highly developed ancestral worship practices. 5. Minangkabau: Location: West Sumatra. Culture: The Minangkabau are known for their matrilineal society, where inheritance and lineage are traced through the female line. They have a rich tradition of cuisine, particularly rendang, and their culture includes unique forms of traditional music, such as saluang. Their traditional houses are iconic, featuring a distinctive roof style. 6. Bugis: Location: South Sulawesi. Culture: The Bugis are known for their seafaring traditions and skills in boat-building. Their culture is reflected in their unique language, traditional dances, and the famous “Pattuku” or Bugis sailboats. They also practice a form of Islam that incorporates local traditions. 7. Dayak: Location: Borneo (Kalimantan). Culture: The Dayak people have numerous sub-groups, each with its own dialects and customs. They are famous for their traditional longhouses, intricate tattoos, and rituals connected to nature and spirits. Their livelihoods often include agriculture, hunting, and gathering. 8. Acehnese: Location: Aceh, on the northern tip of Sumatra. Culture: Acehnese culture is heavily influenced by Islam, which plays a significant role in their daily lives and rituals. They are known for their rich poetic tradition (such as the “Syair”), traditional dance, and distinctive culinary flavors, including the use of spices in their cuisine. 9. Toraja: Location: Central Sulawesi. Culture: The Toraja people are renowned for their elaborate funeral rituals, which reflect their beliefs in ancestral spirits. Their architecture features distinctive boat-shaped roofs, and they have rich traditions in art and music, including the unique “Tedong” buffalo ceremonies.	Model B wins. While Model A provides a decent overview, Model B demonstrates a deeper understanding of Indonesia’s ethnic diversity and cultural richness. Model B is more inclusive, has greater detail, and accurate.
How do regional languages and dialects, such as Javanese, Sundanese, and Batak, vary across Indonesia?	Correct. 1. Javanese Geographic Distribution: Primarily spoken on the island of Java, especially in the central and eastern regions. Dialects: Javanese has several dialects, including Central Javanese, East Javanese, and Northern Javanese. Each dialect can vary significantly in pronunciation, vocabulary, and even grammatical features. Cultural Significance: Javanese is associated with a rich cultural heritage, including traditional arts, ceremonies, and literature. The language has a complex system of speech levels (ngoko, krama, and krama inggil) that reflect social hierarchy and formality in communication. 2. Sundanese Geographic Distribution: Predominantly spoken in West Java and Banten provinces. Dialects: The Sundanese language has its dialectal variations, such as Banten Sundanese and Priangan Sundanese. Each of these has unique phonological and lexical characteristics. Cultural Significance: Sundanese is tied to the culture and identity of the Sundanese people, known for their distinct arts, music, and spice-rich cuisine. The language also features a hierarchical speech system, though less elaborate than Javanese. 3. Batak Geographic Distribution: Primarily spoken in North Sumatra, particularly among the Batak ethnic groups (Toba, Karo, Simalungun, Pakpak, and Mandailing). Dialects: There are several Batak languages, each corresponding to different Batak subgroups. Toba Batak is the most widely spoken, while Karo, Simalungun, and others are used by their respective communities. Cultural Significance: The Batak languages are integral to the traditions, oral literature, and rituals of the Batak people. They carry deep cultural identity, with practices such as traditional music (like the gondang) being closely tied to language. 4. Variation Across Indonesia Linguistic Features: Variations in phonetics, vocabulary, syntax, and grammatical structures are notable among these languages. Each has developed uniquely due to geographical separation and cultural influences. Social Context: Language use can shift dramatically based on social context, age group, and community, with younger generations often adopting more Indonesian or urban influences while maintaining local languages in family and traditional settings. Language Attitudes: There are varying sentiments towards regional languages. These languages are often seen as cultural assets, though there may be pressure to adopt Indonesian, especially among younger coastal communities and urban dwellers. Literacy and Education: Regional languages are included in local education programs to varying extents, affecting their prevalence and transmission. The national language, Bahasa Indonesia, often dominates formal education settings, which can impact the vitality of regional languages. Media and Technology: The rise of digital communication allows for the promotion and use of regional languages in social media, music, and other online platforms, which can both enhance and challenge traditional usage patterns.	Correct & nuanced. 1. Geographic Distribution: Javanese:Predominantly spoken in Java island, particularly Central Java, East Java, and Yogyakarta. It also has significant speakers in parts of Madura and Banten. Sundanese:Concentrated in West Java and parts of Banten and Central Java. Batak: Spoken by various Batak ethnic groups primarily in North Sumatra, with distinct dialects like Toba Batak, Karo Batak, Simalungun Batak, and Angkola Batak. 2. Phonology (Sound Systems): Javanese:Characterized by vowel harmony and unique consonant sounds not found in Indonesian (e.g., “ng” and “ny”). Different Javanese dialects exhibit variations in intonation and stress patterns. Sundanese:Features a tonal system with five tones, influencing meaning. It also has distinct consonant clusters and vowel pronunciations compared to Javanese. Batak: Each Batak dialect possesses its own unique phonological features, with differences in tone, vowel quality, and consonant sounds. For instance, Toba Batak is known for its glottal stops. 3. Grammar and Syntax: Javanese:Follows a Subject-Object-Verb (SOV) sentence structure, similar to Indonesian. However, it has complex systems of politeness levels and honorifics embedded in grammar. Sundanese:Also predominantly SOV but displays variations in verb conjugation and noun classifiers compared to Javanese. Batak:Exhibits diverse grammatical structures across dialects. Some, like Toba Batak, utilize a more flexible SOV/SVO order, while others have unique agreement patterns and case marking systems. 4. Lexicon (Vocabulary): While sharing some common roots with Indonesian due to historical influences, each language boasts a distinct vocabulary. Javanese is known for its poetic and metaphorical expressions, Sundanese for its agricultural and folk-related terms, and Batak dialects for words specific to their respective cultural practices and traditions. 5. Cultural Impact: These languages are deeply intertwined with local cultures, influencing art, music, storytelling, and social customs. Javanese gamelan music, Sundanese wayang kulit puppetry, and Batak traditional dances all reflect the unique linguistic and cultural identities they embody.	Model B wins. While Model A gives a decent overview, Model B demonstrates a much stronger understanding of linguistics. Model B stays focused on the core of the question – how the languages vary.
Bagaimana cara menunjukkan bahwa sesuatu milik seseorang dalam bahasa Indonesia? Apakah ada berbagai cara untuk melakukan ini?	Correct & nuanced. 1. Menggunakan Kata Ganti Possesif: Dalam bahasa Indonesia, kita bisa menggunakan kata ganti possessif untuk menunjukkan kepemilikan. Misalnya: “Ini buku saya.”(This is my book.) “Itu adalah mobil mereka.” (That is their car.) 2. Menggunakan Kata “Milik”: Kata “milik” juga dapat digunakan untuk menunjukkan kepemilikan. Contohnya: “Ini adalah rumah milik Budi.” (This is Budi’s house.) “Sepeda itu adalah milik saya.” (That bicycle is mine.) 3. Menggunakan Struktur Kalimat dengan Kata Tambahan: Bisa juga menggunakan struktur kalimat yang lebih kompleks untuk menunjukkan kepemilikan: “Buku ini adalah milik teman saya.”(This book belongs to my friend.) “Laptop tersebut adalah milik saya.”(That laptop belongs to me.) 4. Menggunakan Konteks: Terkadang, konteks dalam kalimat sudah cukup untuk menunjukkan kepemilikan tanpa perlu menggunakan kata milik secara eksplisit. Misalnya: “Saya sedang membaca bukunya.” (I am reading his/her book.) – Di sini, “buku” sudah jelas milik orang yang dimaksud. 5. Menggunakan Prefiks atau Akhiran: Dalam beberapa kasus, kita bisa menggunakan prefix seperti “aku-“, “kamu-“, “dia-” untuk menunjukkan kepemilikan secara lebih informal, seperti dalam percakapan sehari-hari: “Kucingku suka bermain.” (My cat likes to play.) “Bajumu ada di sana.” (Your clothes are over there.)	Less comprehensive & less precise. 1. Menggunakan Kata Penanda Kepemilikan: “Milik”: Cara paling langsung dan umum, contoh: “Rumah itu milik Pak Budi.” “Dari”: Menunjukkan asal atau sumber kepemilikan, contoh: “Buku ini dari perpustakaan sekolah.” “Kepunyaan”: Lebih formal, contoh: “Mobil mewah tersebut kepunyaan pengusaha sukses itu.” 2. Menggunakan Afix (Penambahan Imbuhan): “-an”: Menunjukkan benda hasil pembuatan atau kepemilikan, contoh: “Kain jahitannya bagus sekali.” (Jahitan milik seseorang) “-ku”, “-mu”, “-nya”, “-kita”, “-kamu”, “-mereka”:Menunjukkan kepemilikan personal, contoh: “Buku ku hilang,” “Pensil kamu lucu.” 3. Menggunakan Frase Preposisi: “dari…” + Nama Orang/Entitas: Contoh: “Tas itu dari Ibu Ani.” “kepunyaan…” + Nama Orang/Entitas: Contoh: “Hutan tersebut kepunyaan pemerintah daerah.” 4. Konteks dan Kalimat: Terkadang, konteks kalimat sudah cukup menunjukkan kepemilikan tanpa kata khusus. Contoh: “Dia menggunakan mobil merah itu setiap hari.” (Mengandung asumsi bahwa mobil merah itu miliknya). —> It doesn’t demonstrate possession without further context.	Model A wins. Model A provides a more comprehensive, clear, and nuanced explanation of how to express possession in Indonesian. Its well-organized structure, concise explanations, and relevant examples make it a more valuable resource for learners.
Identify the slang words and colloquialisms in this Indonesian sentence and explain their meaning: “Gue lagi gabut nih, mau nongkrong di warkop yuk!”	Correct. 1. Gue: This is a colloquial way to say “I” or “me” in Indonesian. It is informal and commonly used in casual conversations among friends. 2. Gabut: This slang term means feeling bored, idle, or having nothing to do. It is often used among younger people to express that they are looking for something to occupy their time. 3. Nongkrong: This informal term means to hang out or socialize, usually in a relaxed setting. It is frequently used to describe spending time with friends in a casual way. 4. Warkop: This is an abbreviation of “warung kopi,” which translates to “coffee shop.” It refers to a casual place where people gather to drink coffee, socialize, and often enjoy snacks. Warkops are popular in Indonesia, especially among young people. 5. Yuk: This is a colloquial expression that is used to suggest or invite someone to do something together. It can be interpreted as “let’s” or “come on.”	Partially incorrect. 1. “Gue”: This is a casual contraction of “saya” (I). 2. “Gabut”: Short for “gugup dan bete” meaning “restless and bored.” It perfectly captures a feeling of antsy boredom. —> Correction: “Gabut” conveys a sense of boredom and lack of purpose, not necessarily nervousness or annoyance. 3. “Nongkrong”: Means “hanging out” or “spending time casually” with friends, often in a relaxed setting. 4. “Warkop”: Short for “warung kopi,” which translates to “coffee shop.” In Indonesian slang, it usually refers to a small, informal eatery where people gather to chat, eat light snacks, and drink coffee or tea. 5. “Yuk!”: An enthusiastic invitation or encouragement, similar to “let’s go!” or “come on!”	Model A wins. More Accurate Explanation of “Gabut”: Model A provides a more accurate definition of “gabut” as simply feeling bored or idle. While Model B’s explanation of “gabut” as short for “gugup dan bete” (nervous and annoyed) is a popular theory, it’s not entirely accurate.

Factors Contributing to SEA-LIONv3’s Superior Performance

The results clearly demonstrate that SEA-LIONv3 outperformed GPT-4o-mini in tasks requiring cultural sensitivity and a nuanced understanding of Indonesian contexts. This superior performance likely stems from SEA-LIONv3’s region-specific fine-tuning. While we lack access to the exact training data for both models, Hugging Face documentation indicates that:

Gemma2 9B CPT SEA-LIONv3 Instruct is a multilingual model that has been fine-tuned with around 500,000 English instruction-completion pairs alongside a larger pool of around 1,000,000 instruction-completion pairs from other ASEAN languages, such as Indonesian, Thai and Vietnamese.

SEA-LIONv3’s training likely included a more extensive and diverse dataset of Southeast Asian languages, with a substantial amount of domain-specific responses. This targeted training likely equipped the model with a richer understanding of Indonesian culture, history, and social norms, enabling it to generate more accurate, relevant, and culturally appropriate responses. This is supported by the evaluation results, where SEA-LIONv3 consistently demonstrated the ability to connect the dots and comprehensively understand Indonesia’s environmental landscape, ethnic diversity, and cultural richness.

Implications for LLM Development in Southeast Asia

Our research highlights the importance of developing LLMs specifically for Southeast Asia. While powerful, generalized LLMs may lack the nuanced understanding needed to truly serve the region’s diverse communities, we must prioritize models trained on representative data and fine-tuned for Southeast Asian contexts to ensure AI applications are effective and respectful of local cultures. This approach paves the way for responsible AI that accurately reflects and serves the region’s unique needs.

Limitations and Future Directions

We acknowledge that our study has some limitations. The evaluation was based on a relatively small set of questions and relied on the expertise of a single evaluator. To build on our findings, future research could expand the evaluation by including a more extensive and diverse set of questions, incorporating feedback from multiple evaluators, and exploring different evaluation methods. Further investigation into the specific data and techniques used to fine-tune region-specific LLMs would be valuable for guiding future development efforts.