This course introduces technologies for language processing, with an emphasis on modern approaches based on large language models (LLMs). Students will learn the foundational neural network methods widely adopted in natural language processing today, including the transformer architecture underlying LLMs, as well as the pre-training, post-training, and alignment stages in the development of LLMs. The course also covers selected traditional natural language processing methods from a linguistic perspective. The course focuses on the theoretical foundations of the methods and algorithms, while practical exercises provide students with valuable hands-on experience.
Learning Outcomes
Upon successful completion, students will have the knowledge and skills to:
- Explain, implement, and experiment with embedding-based text representations and their application in downstream NLP tasks.
- Explain, implement, train, and evaluate commonly used neural network architectures for language processing, including recurrent neural networks and transformer models.
- Explain the principles underlying large language models (LLMs), including training objectives, architectures, fine-tuning strategies, and critically assess their capabilities and limitations in applied settings.
- Explain and evaluate information retrieval methods and how they are combined with LLMs for retrieval-augmented generation.
- Design and assess solution pipelines for language processing tasks such as text classification using methods and techniques covered in the course.
Indicative Assessment
- Quizzes (20) [LO 1,2,3,4]
- Assignments (30) [LO 1,2,5]
- Final Examination (50) [LO 1,2,3,4,5]
The ANU uses Turnitin to enhance student citation and referencing techniques, and to assess assignment submissions as a component of the University's approach to managing Academic Integrity. While the use of Turnitin is not mandatory, the ANU highly recommends Turnitin is used by both teaching staff and students. For additional information regarding Turnitin please visit the ANU Online website.
Workload
Lectures, laboratory sessions and self study to a total of 130 hours
Inherent Requirements
No specific inherent requirements have been identified for this course.
Requisite and Incompatibility
Prescribed Texts
The following reference books will be used.
- Introduction to Information Retrieval, C.D. Manning, P. Raghavan and H. Scutze, Cambridge University Press, 2008.
- Foundations of Statistical Natural Language Processing, C.D. Manning and H. Scutze, MIT Press, 1999.
Assumed Knowledge
Programming ability in C, C++, Java or Python, knowledge in probability theory, calculus, and linear algebra, at an undergraduate level.
Fees
Tuition fees are for the academic year indicated at the top of the page.
Commonwealth Support (CSP) Students
If you have been offered a Commonwealth supported place, your fees are set by the Australian Government for each course. At ANU 1 EFTSL is 48 units (normally 8 x 6-unit courses). More information about your student contribution amount for each course at Fees.
- Student Contribution Band:
- 2
- Unit value:
- 6 units
If you are a domestic graduate coursework student with a Domestic Tuition Fee (DTF) place or international student you will be required to pay course tuition fees (see below). Course tuition fees are indexed annually. Further information for domestic and international students about tuition and other fees can be found at Fees.
Where there is a unit range displayed for this course, not all unit options below may be available.
| Units | EFTSL |
|---|---|
| 6.00 | 0.12500 |
Course fees
- Domestic fee paying students
| Year | Fee |
|---|---|
| 2027 | $5640 |
- International fee paying students
| Year | Fee |
|---|---|
| 2027 | $7200 |
Offerings, Dates and Class Summary Links
ANU utilises MyTimetable to enable students to view the timetable for their enrolled courses, browse, then self-allocate to small teaching activities / tutorials so they can better plan their time. Find out more on the Timetable webpage.
Class summaries, if available, can be accessed by clicking on the View link for the relevant class number.
Second Semester
| Class number | Class start date | Last day to enrol | Census date | Class end date | Mode Of Delivery | Class Summary |
|---|---|---|---|---|---|---|
| 10086 | 26 Jul 2027 | 02 Aug 2027 | 31 Aug 2027 | 29 Oct 2027 | In Person | N/A |