AI Singapore (AISG) is a national initiative driven by the Ministry of Digital Development and Information (MDDI), the Infocomm Media Development Authority (IMDA), and the National Research Foundation, Singapore (NRF).
AISG brings together Singapore-based research institutions and the broader ecosystem of AI start-ups and companies to support impactful research, develop talent and power Singapore's AI efforts.
For more information on AI Singapore, please visit https://www.aisingapore.org.
This position will be hosted at the Nanyang Technological University (NTU) under VP (Artificial Intelligence & Digital Economy)’s office and we welcome you to join our community.
Duties and Responsibilities:
- Develop and maintain evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs).
- Keep up to date and experiment with the latest research in multilingual, multicultural, and multimodal LLM evaluations such as the LLM-as-a-Judge paradigm.
- Work with partners to collect, translate and verify evaluation datasets.
- Perform the necessary data preparation and analysis, AI modelling, coding, testing, validation and deployment to ensure reliable and scalable AI solutions.
- Collaborate with cross-functional teams within AI Products to design and resolve issues.
- Maintain code repository and documentation standards.
- Contribute to community engagement activities such as sharings via technical session meet-ups and article write-ups, and participating in discussion forums.
Requirements:
- Graduate from a degree program in computer science, AI, data science or related fields (or equivalent practical experience).
- Deep understanding of LLM evaluation experimental design and the advantages/limitations of various LLM evaluation methods.
- Experience in LLM inference frameworks such as vLLM and AI/Deep Learning frameworks such as PyTorch.
- Writing production level code in Python and using version control systems such as Git.
- Strong written and verbal communication skills.
- Independent learner that is capable of reading and understanding research papers.
- Fluent in English and one other Southeast Asian language for the purposes of understanding how to build quality evaluations from a multilingual and multicultural perspective.
We regret that only shortlisted candidates will be notified.
Hiring Institution: NTU
