Tag: english-articles

Lessons from the Text-to-Text Transformer (T5) ablation studies

Reading the T5 paper was a pleasure for me and has helped me learn a great deal.

The paper was written in a way that’s easy to understand and follow. It used a long-format style (44 pages) and this has allowed the authors to explain things in detail.

And most importantly, its explicit focus was to do ablation studies to shed a clear light on what works and what doesn’t, pointing the way for future explorations.

A quick intro to T5

  • The framing as text-to-text enables it to solve both generation and classification problems using an exact same encoder-decoder architecture, and without the need for using different “heads” for different problems (like what was used with BERT).  This is a really cool and ambitious problem modelling. To instruct the model on what to perform, a prefix is added to the input to signal what’s expected in the output (Ex: “translate English to German: <input>”)
  • In pretraining, the dropped-out tokens can be phrases (multiple continguous words); vs BERT: dropping out single words.

Below is a non-exhaustive list of key ablation studies in the paper. Continue reading Lessons from the Text-to-Text Transformer (T5) ablation studies

A Transformer model for inserting Vietnamese accent marks

Updates:  The trained model + instructions to use can now be downloaded from HF here.

In this post, I summarize how I made use of Huggingface’s transformer library to re-solve an NLP problem related to the Vietnamese language.

The problem

After learning about Hidden Markov models about 10+ years ago, I decided to apply it to building a small, but practical, toy that can auto insert accent marks for Vietnamese language.

In a nutshell, Vietnamese has some letters that have additional marks put on them. For ex, in addition to the letter ‘a’, the Vi alphabet also contains these “marked versions”: ă, â.

And for each of these 3  versions (a, ă, â), we can then put the 5 tones on them. An example for ‘ă’ will be:  ắ (acute),  ằ (grave), ẳ (hook), ẵ (tilde), ặ (dot).

Continue reading A Transformer model for inserting Vietnamese accent marks

A few notes on Items Response Theory (IRT) and Computerized adaptive testing

Recently, I was thinking about how to improve the accuracy of assessment tests for ESL learners and so I googled and found Computerized Adaptive Testing (CAT).

During the process, I accidentally discovered an interesting theory behind it. It’s called Items Response Theory or IRT for short.

So I’ve spent some time reading up about it and in the process, picked up a few very useful bits about statistical hypothesis testing, which I’m very glad to have learned.

Below, I share the most important ideas about IRT that I’ve learned.

Continue reading A few notes on Items Response Theory (IRT) and Computerized adaptive testing

Interesting points from “The best way to learn Math facts”

I recently happened to come across an article on the topic of learning Math facts, titled “Fluency Without Fear: Research Evidence on the Best Ways to Learn Math Facts” by Jo Boaler.

The topic of making Math cool again is, in my opinions, very critical b/c it’s the foundation of all sciences and it should have been cool. The way it’s currently taught and approached makes it uninteresting and scary.

Below are a couple of interesting points I gathered from the above article:

Continue reading Interesting points from “The best way to learn Math facts”

2 interesting points from Andrew Ng’s interview with Geoffrey Hinton

In this interview in a Coursera course by Andrew Ng with Geoffrey Hinton, who according to Ng is one of the “Godfathers of Deep learning”, I found 2 points that were quite interesting and thought-provoking.

On research direction

When asked about his advice for grad students doing research, Hinton said, at about 30 mins in:

Continue reading 2 interesting points from Andrew Ng’s interview with Geoffrey Hinton

“Yes, I have”: a story about an encounter with Itzhak Perlman

This is a story about an encounter of a guest with Itzhak Perlman, as told by Matthew Kelly in his book The Rhythm of Life.

The story begins:

Continue reading “Yes, I have”: a story about an encounter with Itzhak Perlman