english-articles Archives

Lessons from the Text-to-Text Transformer (T5) ablation studies

Reading the T5 paper was a pleasure for me and has helped me learn a great deal.

The paper was written in a way that’s easy to understand and follow. It used a long-format style (44 pages) and this has allowed the authors to explain things in detail.

And most importantly, its explicit focus was to do ablation studies to shed a clear light on what works and what doesn’t, pointing the way for future explorations.

A quick intro to T5

The framing as text-to-text enables it to solve both generation and classification problems using an exact same encoder-decoder architecture, and without the need for using different “heads” for different problems (like what was used with BERT). This is a really cool and ambitious problem modelling. To instruct the model on what to perform, a prefix is added to the input to signal what’s expected in the output (Ex: “translate English to German: <input>”)
In pretraining, the dropped-out tokens can be phrases (multiple continguous words); vs BERT: dropping out single words.

Below is a non-exhaustive list of key ablation studies in the paper. Continue reading Lessons from the Text-to-Text Transformer (T5) ablation studies

A Transformer model for inserting Vietnamese accent marks

Huggingface’s transformer library is enabling engineers and developers to access the latest latest developments in AI research. Kudos to them.

Below, I summarize how I made use of their library to re-solve an NLP problem related to the Vietnamese language.

The problem

After learning about Hidden Markov models about 10+ years ago, I decided to apply it to building a small, but practical, toy that can auto insert accent marks for Vietnamese language.

In a nutshell, Vietnamese has some letters that have additional marks put on them. For ex, in addition to the letter ‘a’, the Vi alphabet also contains these “marked versions”: ă, â.

And for each of these 3 versions (a, ă, â), we can then put the 5 tones on them. An example for ‘ă’ will be: ắ (acute), ằ (grave), ẳ (hook), ẵ (tilde), ặ (dot).

Continue reading A Transformer model for inserting Vietnamese accent marks

A few notes on Items Response Theory (IRT) and Computerized adaptive testing

Recently, I was thinking about how to improve the accuracy of assessment tests for ESL learners and so I googled and found Computerized Adaptive Testing (CAT).

During the process, I accidentally discovered an interesting theory behind it. It’s called Items Response Theory or IRT for short.

So I’ve spent some time reading up about it and in the process, picked up a few very useful bits about statistical hypothesis testing, which I’m very glad to have learned.

Below, I share the most important ideas about IRT that I’ve learned.

Continue reading A few notes on Items Response Theory (IRT) and Computerized adaptive testing

A few thoughts on learning, responsibility and commitment

Up to recently, I only knew of one meaning of “learning”: that is to take in more knowledge, or to improve one’s existing knowledge.

So it’s mostly about information and knowledge. About what one knows.

But with time and more experience in management, communication and work in general, I realized that there’s another type of learning that is even more important for one to make progress in work and life.

And it’s not quite the above type of learning more knowledge.

Continue reading A few thoughts on learning, responsibility and commitment

Building a culture of high performance: Learn to give and receive feedback well

There are many things we can learn from the book Netflix’s Culture of reinvention. Among them, a practice that we can all learn and apply is its insistence on “selfless candor”: the practice of improving performance through receiving regular feedback (from everyone).

To build a culture that really embraces constant learning and improvements, learning to give and receive feedback well is a sine qua non.

Without constant 360-degree feedback, we identify our mistakes more slowly (and sometimes completely oblivious to our mistakes) and as a result, we learn and improve more slowly.

Continue reading Building a culture of high performance: Learn to give and receive feedback well

Lessons from Netflix’s culture of reinvention

The new book about Netflix’s culture of reinvention is superb, in my opinions. Its full title is: “No Rules Rules: Netflix and the Culture of Reinvention” by Reed Hastings (CEO) and Erin Meyer.

The book shares how Netflix has built a culture of reinvention, whereby making it one of the leaders in the creative business.

This book contains many things that are, well, to use Netflix’s lingo, “stunning”.

Continue reading Lessons from Netflix’s culture of reinvention