How Does an Algorithm Align my Teeth?

Darla in Finding Nemo

This year, I finally decided to get my teeth straightened.

I had worn removable aligners as a kid, but given my lack of will-power, I did not wear them consistently enough and my teeth never got straight 100%. This has been bothering me for the last 10+ years (#firstworldproblems).

So, in December 2020, I had my first appointment at the dentist to make all the scans and begin an Invisalign treatment. Invisalign is a custom-made orthodontic system for your teeth, consisting of a set of clear “invisible” and removable aligners. Each week, you receive a new set that gradually helps move your teeth into the desired place. Since I did not want to get old-school braces and look like a 30-year-old Darla in Finding Nemo, Invisalign seemed like a good option.

After the second appointment in January, I walked home with my first set of clear aligners in my teeth, designer-looking Invisalign package in my hands, and disconcertment in my spirit, due lack of information. How does this technology work? How is the teeth trajectory calculated? How accurate is it? While I love my dentist (because he is competent, NOT because he happens to be quite handsome), I did not really get any helpful background information, except for a beautifully marketed Invisalign flyer, informing me that “A lot of powerful technology and doctor’s expertise combine to make a digital plan for shaping your new smile.” 1https://www.invisalign.com/how-invisalign-works/treatment-plan What a great marketing pitch. But what exactly is the “powerful technology” behind it? I decided to investigate a bit.

The Technologies

There is not one technology behind Invisalign, but several. The treatment starts with portrait pictures of your smile, which serve as basis for a 2D facial analysis (or “smile” analysis). I will talk about this step a bit more in the next section.

The dentist will then make a scan of your teeth, using the iTero element® scanner. It’s a fancy name for a 3D scanner, based on 3D reconstruction technology coming from the fields of computer vision and computer graphics.2Fun fact: Disney is behind some of the research in teeth reconstruction. Perhaps they want their employees to have as straight teeth as their movie characters? https://la.disneyresearch.com/publication/model-based-teeth-reconstruction/ This technology is based on 2D images: for Invisalign, a hand-held scanner is used, literally stuck in your mouth and moved around. This method is called intraoral digital dental impression scan.

Picture from Wu et al.’s ACM SIGGRAPH Asia 2016 presentation 3https://pdfs.semanticscholar.org/90e0/b93e049509bf9058206cce945a4cc2fcc150.pdf

It replaces the old-school mold that they used to stick into your mouth, making you gulp for air (though it has been shown to be less accurate than digital impressions made from such casts4See Flügge et al., 2013).

Do they still torture people with those? 5 Picture from https://www.dentalproductsreport.com/view/10-golden-rules-taking-impressions

The method used by the scanner is called parallel confocal imaging, another fancy word, which means that thousands of points of laser light are used to trace, capture, and reconstruct the shape of your teeth.

Doing all of this is not a trivial step: the technology needs to accurately capture your teeth geometry and the gingiva (the junction between your gums and teeth), to be used as basis for planning the treatment plan. It also needs to “guess” where your roots are, since they are not visible. According to a 2003 paper6See Beers et al., 2003 on Invisalign, this step is (or was) done, using simple statistics7A linear model, likely a regression, is fit using four teeth properties, and a cost function is used to find the best fitting model..

I say this step was done this way, as I imagine newer methods are used these days, possible using more advanced methods, that take into account more properties of your teeth and mouth. In the picture below, for instance, you can see a method from 2016, which uses image recognition on a simple picture (no 3D scan), to trace your teeth. Clearly things have been moving since 2003.

Picture from Wu et al.’s ACM SIGGRAPH Asia 2016 presentation8https://pdfs.semanticscholar.org/90e0/b93e049509bf9058206cce945a4cc2fcc150.pdf

So, the first important step is to create an accurate model of your teeth.

Next, an algorithm is used to calculate the perfect position of your teeth and the difference between the current and perfect state. This is done (according to Invisalign) using their ClinCheck® software. What is behind this software? Invisalign’s website tells us, “The algorithm helps calculate just the right amount of force for every tooth movement”, “The software, with input from your doctor, helps ensure every tooth moves in the right order and at the right time” and the whole thing is “Powered by data from 9 million smiles.” Let’s dig a bit deeper.

“Beauty is in the eye of the beholder” – but who exactly is the beholder?  

The first important question to ask is: How does whoever programmed the software (or even the dentist) know, what the teeth formation should ideally look like? What is the “right” amount of movement? What makes the teeth aesthetically pleasing? Zimmerman and Mehl (2015) write in their survey:

“Redesigning the appearance of the anterior [front] teeth in the smile design process is a demanding task, first, because this changes the patient’s smile characteristics and, second, because the esthetic wishes of the patient must be accommodated within the predetermined functional, structural and biological framework.”

Sounds quite complicated. So how does Invisalign accomplish this?

The paper I mentioned from 2003, which gives details on the Invisalign process, does not give a very satisfying answer to this question, except saying that the teeth are placed in “a clinically and esthetically acceptable final position.”9 see Beers et al., 2003 Creating the perfect smile seems to not be an exact science – according to another paper from 2006, “Many scientific and artistic principles considered collectively are useful in creating a beautiful smile.” 10See Davis, 2007 Let’s have a look at some examples.

Aesthetic Principles

As mentioned earlier, my treatment started with a picture of my smile. To be honest, I originally thought there is some trivial reason for this – perhaps they liked my face and have some sort of internal hall of fame for best smiles achieved. Well, likely that is not the reason, because while researching I came across something called Smile Design. These days, there are many guidelines to achieving a “perfect” smile, and profile pictures provide valuable input to achieve a person’s smile goals.

Bottom is deemed as more aesthetically pleasing, as the gums are not showing (picture from Davis, 2007).

There are several available systems that have been developed for this purpose, one of which is called Digital Smile System (DSS). A look on their website confirmed that they teamed up with Invisalign.11https://digitalsmiledesign.com/planning-center/dsd-and-invisalign-tps

Four main parts are likely analysed from the pictures: the lipline, the “smile curve,” the shape of your teeth, and certain facial features, such as height and width.

Your teeth midline should be centered, as a distorted midline is deemed unaesthetic (picture from Davis, 2007)

Scientific Principles

Of course, your smile is not the only thing that is important. By creating a 3D model of your teeth, roots, and gums, the system can also determine what is feasible and what is not. For instance, teeth can only move two ways, by translation or rotation (or a combination). Furthermore, force needs to be applied by taking into account the center of resistance of a tooth. This is similar to the center of gravity from physics: every object has one point, at which it can be balanced, and movement depends on the line in which force is exercised to this point.

My boyfriend and I, trying to move the center of gravity of the tower of Pisa. In case you every wondered why the thing is not falling, knowing about centers of resistance should help you understand.12 https://leaningtowerpisa.com/facts/why-pisa-leaning-tower-does-not-fall

To summarise: Invisalign uses aesthetic and scientific principles, just like an orthodontist would, and likely examples of previous outcomes (given they have data from 9 million smiles) to create a realistic “goal” model of your teeth.

Movement Planning

Once we have a model of the existing teeth and a model of the ideal teeth position, it is time to calculate how to get from A to B. The process is called Movement Planning. Movement planning is done, by taking into account how force should be applied, to make each tooth move in the desired way. Importantly, Invisalign aligners can only push, not pull, which needs to be taken into account.

Example of a initial and goal model of teeth.13 image taken from https://perfect-smile4u.de/blog/blick-in-die-zukunft-in-echtzeit-clincheck-pro-von-invisalign/

Likely, the software calculates several paths of how to get there, and a clinician or the dentist chooses the best path.

A set of intermediate steps are calculated, which the teeth have to reach with the weekly changing aligners. Invisalign’s software outputs the treatment plan with all the intermediate steps, which the dentist can view and modify, if he or she thinks the teeth will not move the desired way. In the end, virtual planning is not the same as actual, physical planning by a dentist. Nevertheless, many companies are trying to minimise the contact with the dentist, to minimise costs, and leave things up to the software.

Using another technology called SmartStage, the set of aligners are then produced and 3D-printed.

Outcome Analysis

The last part of the process is to make another scan of your teeth and compare the result with the desired result (I have not yet reached this point).

Picture taken from Beers et al., 2007. The colours show differences in expected and actual treatment outcome.

If you actually wore your aligners the recommended 20-22 hours per day, your teeth should now be perfect, your smile bright and wonderful, so you can live happily ever after. Right?

The Bad News

Invisalign was introduced in 1999 by the U.S.-based Align Technology, a manufacturer of 3D digital scanners 14Although they are by no means the first to have the idea of an “invisible” aligner; the first was already introduced by Ponitz in 1971. However, the first scientific clinical study to evaluate its effectiveness was not done until 10 years later, in 2009, after a 2005 systematic review showed that no adequately designed studies could be found and therefore no scientifically valid claims could be made on treatment effect of the aligners.

Subsequent studies, such as by Kravitz et al., 2009 showed that there were significant differences between the proposed virtual results and the actual clinical tooth movements, in some cases showing less than 50% accuracy in the movement. One study from 201515See Hennessy & Al-Awadhi, 2015 concludes by saying:

“Clear aligner clinical usage has not been matched by high-quality research.
Until these data become available, these appliances will continue to be viewed with a degree of scepticism by many orthodontists”

A systematic review from 2018 found that only 3 of the 22 surveyed studies showed a low risk of bias, and concluded by saying:

“Despite the fact that orthodontic treatment with Invisalign® is a widely used treatment option, apart from non-extraction treatment of mild to moderate malocclusions of non-growing patients, no clear recommendations about other indications of the system can be made, based on solid scientific evidence.”16See Papadimitriou et al., 2018

The latest research I found, a paper starting with “Has Invisalign improved?” in the title, from 2020,17See Haouili et al., 2020 concludes that it did improve a bit (i.e. now more around 50% accuracy, yay…), but “Despite the improvement, the weaknesses of tooth movement with Invisalign remained the same.”

You can imagine me at this point in my research, aligners in my mouth, nervous twitch on my face, wondering why I did not think of checking all of this before starting the treatment…

…like so.

The Good News

Apparently there were several generations of aligners and the third one got things a bit more right. All of it has to do with how force is applied to your teeth and if that force reaches the center of resistance of each tooth, which is the only way to achieve movement. Removable aligners cannot reach this center of resistance easily, so attachments came to the rescue. Attachments are basically like clear versions of regular brackets, that are glued to your teeth, which nowadays are automatically added by the software in certain cases, like derotations and root movements.

Picture of the teeth of a Reddit user, with attachments on each tooth.

The funny thing of course, is that if you need attachments on every tooth, there is no more big difference between regular brackets and Invisalign in terms of aesthetics, since nowadays regular fixed ceramic brackets can be quite “invisible” as well (no more metal toolbox in your mouth).

But for some more good news: Most papers seem to agree that “light” cases can be successfully treated with Invisalign, and the dentist should also let you know from the beginning, if your case is treatable using this system, or requires the old-school brackets. Since I have the Invisalign Lite version (only minor changes to the front teeth), I should be a very light case (I am more talking to myself, at this point, to help me feel confident).

I recently had to go back to the dentist to get my attachments added. While the first few weeks were easy-peasy, the attachments take some getting used to. I will update this article, once my treatment is complete and add my own experience.

*** UPDATE ***

I am done with my treatment and the outcome is impressive! So, forget everything I said before. Just kidding, here are a few take-aways:

  1. Teeth tech is cool.
  2. It’s all about attachments — without those, things will not move the way they should, and of course Invisalign’s ads will not show you that attachments are not as comfy-invisible as just trays.
  3. After your treatment, you might need an after-treatment of a few weeks (I needed 5 extra weeks) to fix small last issues, which prolongs the whole endeavour.
  4. After the after-treatment, in order to prevent your teeth from going back to before, you need to a) wear a last pair of trays over night for life, or b) get some wires put in for life. I chose the latter and can tell you, they take some time getting used to (it probably took me 4 weeks to stop being aware they are in my mouth).
  5. Is it worth it? Absolutely. I am extremely happy with the outcome and have received compliments on my beautiful straight teeth. 🙂

References

Beers, A., Choi, W., & Pavlovskaia, E. (2003). Computer-assisted treatment planning and analysis. Orthodontics & craniofacial research, 6 Suppl 1, 117-25 .

Flügge, T.V., Schlager, S., Nelson, K., Nahles, S., & Metzger, M. (2013). Precision of intraoral digital dental impressions with iTero and extraoral digitization with the iTero and a model scanner. American journal of orthodontics and dentofacial orthopedics : official publication of the American Association of Orthodontists, its constituent societies, and the American Board of Orthodontics, 144 3, 471-8 .

Haouili, N., Kravitz, N., Vaid, N.R., Ferguson, D., & Makki, L. (2020). Has Invisalign improved? A prospective follow-up study on the efficacy of tooth movement with Invisalign. American journal of orthodontics and dentofacial orthopedics : official publication of the American Association of Orthodontists, its constituent societies, and the American Board of Orthodontics.

Hennessy, J., & Al-Awadhi, E. (2016). Clear Aligners Generations and Orthodontic Tooth Movement. Journal of Orthodontics, 43, 68 – 76.

Lagravère MO, Flores-Mir C. The treatment effects of Invisalign orthodontic aligners: a systematic review. J Am Dent Assoc. 2005 Dec;136(12):1724-9. doi: 10.14219/jada.archive.2005.0117. PMID: 16383056.

Liping Zheng; Guangyao Li; Jing Sha (2007). “The survey of medical image 3D reconstruction”. Fifth International Conference on Photonics and Imaging in Biology and Medicine. Proceedings of SPIE. 6534. pp. 65342K–65342K–6. doi:10.1117/12.741321. S2CID 62548928.

Papadimitriou, A., Mousoulea, S., Gkantidis, N., & Kloukos, D. (2018). Clinical effectiveness of Invisalign® orthodontic treatment: a systematic review. Progress in Orthodontics, 19.

Ponitz RJ. Invisible retainers. Am J Orthod 1971; 59: 266 – 272

Kokich, V., Kiyak, H., & Shapiro, P. (1999). Comparing the perception of dentists and lay people to altered dental esthetics. Journal of esthetic dentistry, 11 6, 311-24 .

Kravitz, N., Kusnoto, B., Begole, E., Obrez, A., & Agran, B. (2009). How well does Invisalign work? A prospective clinical study evaluating the efficacy of tooth movement with Invisalign. American journal of orthodontics and dentofacial orthopedics : official publication of the American Association of Orthodontists, its constituent societies, and the American Board of Orthodontics, 135 1, 27-35 .

Smith, R., & Burstone, C. (1984). Mechanics of tooth movement. American journal of orthodontics, 85 4, 294-307.

Zimmermann, Moritz; Mehl, Albert (2015). Virtual smile design systems: a current review. International Journal of Computerized Dentistry, 18(4):303-317.

Siri: Artificially Intelligent or Naturally Stupid?

Artificial Intelligence (AI) is one of those words that is thrown around by everybody and their grandmother these days. If I were to ask you what is AI, what would your answer be? Perhaps you would recall some movies you may have watched, such as Terminator, Her, or Ex Machina, and say AI is one of those evil things that will take over the world and kill people one moment or another.

Or, perhaps, you would take a less apocalyptic, sci-fi approach and say, well, it is human intelligence demonstrated by machines. Next I would ask you, and what is intelligence? Then you would perhaps answer, well, intelligence is a wishy-washy concept (in fact, intelligence has long been studied by psychologists, sociologists, biologists, neuroscientists, or philosophers, leading to over 70 definitions1Legg, S., & Hutter, M. (2007). A collection of definitions of intelligence. In B. Goertzel & P. Wang (Eds.) Proceedings of the 2007 conference on advances in artificial general intelligence: Concepts, architectures and algorithms: Proceedings of the AGI workshop 2006 (pp. 17–17). Amsterdam, The Netherlands: IOS Press. and can mean anything from being able to reason, learn, acquire knowledge to being self-aware or emotionally intelligent (kudos to you for such a sensitive answer!). Then I would ask you, and where have you encountered such an intelligence displayed by machines? At which point you might say “Siri is a bit like this? Or maybe Google?”

Ah, now we get to interesting types of technologies. Siri recognises what you say and formulates an answer, which it does by using Natural Language Processing (NLP), a specific type of Machine learning (ML). ML in turn, is a specific type of AI. And back we are to the question, what is AI? (Fear not, we will get back to NLP and ML shortly).

Come on Siri, leave Kant out of it, you know it’s 422Don’t know what this is referring to? Shame on you. Go read the Hitchhiker’s Guide to the Galaxy and come back to this blog when you finished and are properly educated. And bring a towel!.

One can take a broad view on AI, where pretty much anything based on statistics is a form of AI. Some companies have used such a broad view to their advantage, by marketing their products as “AI-based” even if they are actually built on simpler regression models or rules. This even has its own term, known as “AI washing.”3https://www.thinkautomation.com/bots-and-ai/watch-out-for-ai-washing/

At the other end of the spectrum, you have those who believe AI should only refer to Artificial General Intelligence, something that comes closer to our Sci-Fi Arnold. Let me give you an example of each.

Take Siri, for instance. Siri is currently somewhere in between those two spectrums. What would a simple, rule-based little brother of Siri (let’s call him Arnold) look like? You would take your favourite programming language (mine is Python), and start typing a few lines, like so: if Michaela says “Hello”: then Arnold should respond: “Hello, Michaela!” In Python, it would look something like this:

I can now run this script, type in “hello” and get an answer:

Wow amazing! Although it would also be great, if Arnold knew what to do, if I don’t type exactly the word “hello.”

Will Arnold kill us all soon?

Doesn’t look like it.

This is what you would call a rule-based system (a veeery simple one)4You can try coding this yourself, by going to colab.research.google.com, by the way 🙂. Everything that happens needs to be explicitly coded, otherwise it will generate an error (or go into an endless “No comprendo, José” loop).

Of course, I made it sound highly simplistic, but many companies actually use chatbots based on a rule-type logic and call it “AI-based.” But in terms of intelligence, they might be more similar to a Roomba, one of those cleaning robots that you might have in your home. It will talk at times — but it only talks because it has been instructed to talk at those times, like when returning “home”. It simply checks if a rule applies and follows instructions. No machine learning there and experts in the field would likely not call this AI (although the latest Roomba’s use machine learning to clean, post on that to come!).

Siri is a bit different.

Even though she tells you that she only understands what she has been programmed to understand, it is not like in our example above. Because Siri can kill you.

Just kidding!

Siri first uses speech recognition technology to translate your speech into text. This means that Siri records the words you say into your microphone, sends them to an Apple server, accesses a database, breaks down your words into teeny-tiny units of sounds called phonemes, uses statistics to figure out the probability of your sequence of phonemes to match one that is recorded in the database, to finally decide if there is a “match” or not. Homophones are especially tricky (i.e., did you say merrymarry, or Mary?), so Siri may also analyse the sentence as a whole, break it down into its linguistic parts-of-speech (such as nouns, verbs, adjectives), and figure out which one of the three words is most likely. You see now why Siri is not happy when you speak to her in Swiss-German or some other dialect? How should poor Siri know that Chuchichäschtli means Küchenschrank (German for cupboard)?

Actually, nice researchers are working on trying to build speech recognition systems that recognise your slang-y kinda language (e.g., Swisscom and the Language and Space Lab at UZH).

A day in the near future

Where were we? Right, so now Siri recognised what you said (or not).

Next, she tries to figure out the meaning behind what you said, to know how to proceed. For this she uses Natural Language Understanding (NLU) (a subfield of Natural Language Processing) technology, such as intent classification. Let’s imagine you say “Siri, wake me up at 6am.” After having converted your speech into text, she now uses a machine learning model, trained on a large dataset of sentences with their intents (e.g., for a sentence “Wake me up,” the intent-label might be: “set alarm”. For a sentence “Remind me,” the intent-label might be: “set reminder”), to predict the intent of this sentence.

Once she knows the intent is “set alarm,” or “remind me” she uses something called slot filling. Setting the alarm requires her to know at what time, so she will look for something in the sentence that matches a time. i.e. “6am”. She has been instructed to open the time app on your device and set the alarm to the time you specified. If no time is specified, she will ask you for the missing information. Similarly, for a reminder, she will need to know what the reminder should be.

If you ask her if a duck’s quack can echo, she will recognise that this is a knowledge-type question and that she should query the internet (instead of filling slots). Like so:

You see, a bit more complicated than our “if … then…” statement earlier. And also, much more versatile.

Hm quite evasive…I was more sure about Arnold.

Now we have something that we can safely call AI. Siri “listens” to your speech through your microphone, processes it, learns something from it, and performs an action, all autonomously.5This goes along the line of Russell and Norvig’s definition of Artificial Agents: Russell, S. and Peter Norvig. “Artificial Intelligence – A Modern Approach, Third International Edition.” (2010).

But this is not Artificial General Intelligence (AGI). AGI would imply that Siri would be able to learn how to learn. Right now she still relies on a programmer in the background to provide her with some “if…then…” instructions on how to access which database, and which algorithm to use with which data, to learn something very specific.

Siri agrees.

Instead, AGI would imply that Siri would figure the above out by herself. In fact, natural language understanding belongs to a class of problems that has been termed “AI-hard,”6An analogy to the NP-hard problem from computer science. which means that if you could solve such a complex problem, you would be at the root of the “AI-problem” itself: it would imply Siri could memorise everything that has been said, take context into account, take decisions autonomously, potentially respond with emotions, and therefore, do what a human can do.

Sounds creepy? I agree.

AGI is not yet on the horizon7But, whether you or me like it or not, research is working towards it, i.e. see Pei, J., Deng, L., Song, S. et al. Towards artificial general intelligence with hybrid Tianjic chip architecture. Nature 572, 106–111 (2019). https://doi.org/10.1038/s41586-019-1424-8, and luckily not, because we are currently still far from figuring out what ethics apply when machines are involved and what makes us trust in AIs (I hope to contribute to this line of work in my PhD). This is why you will sometimes see funny experiments like the Moral Machine8https://www.moralmachine.net/, where researchers are interested to see which “lesser evil” people will chose, like killing two passengers or five pedestrians. If we don’t even know what’s right, how should a machine know?

I can see you rolling your eyes at me now, because you made it all the way to this part of the post, are realising now that it is soon ending, and you probably still don’t know what AI is. Let me re-assure you, neither does the research community at times.9Kaplan, A. and M. Haenlein. “Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence.” Business Horizons 62 (2019): 15-25. The answer is, it depends on what you consider as intelligent.

More importantly: will Siri kill us? Here the answer is clearly no.

For now. 🙂