Sign-translation gloves

Three students from Christian Alliance Sun Kei Secondary School — Noddy Chung, Michael Ng and Jacky Chan — built a sign-language translation glove. The same meaning is signed differently from country to country and region to region; to help deaf and mute people communicate with the rest of the world, the three spent three months developing the glove.

Michael explained the principle: when the wearer signs, the glove records the bend, orientation and tilt of the fingers, compares the readings against a phrase database, and outputs text and speech.

Working together, the three collected data for around thirty phrases using the sensors on the glove, checking each reading against the actual finger positions so that the wearer would be understood more easily.

The report set the project against the government's push on innovation and technology in schools, and closed on the hope that student and university research alike would move out of the exhibition hall and into everyday use.

The video cut of the same interview, published to unwire's podcast channel on the day. It runs through the demonstration on camera — the glove goes on, a sign is made, and the phone speaks — with some footage drawn from the local sign-language programme Sign Language from the Deaf Stars.

In the middle of last year two students at the University of Washington developed SignAloud, a pair of sign-language translation gloves that work with a computer: put them on, and signs are matched against a database and turned directly into speech and text. Less than a year later, secondary-school students in Hong Kong have taken the idea further. Three Form 5 students from Christian Alliance Sun Kei Secondary School — Noddy Chung, Jacky Chan and Michael Ng — have made a portable sign-language translation glove that translates with a phone app, so that deaf and mute people can communicate with those around them.

Built for what signers actually need

The same meaning is signed differently in different parts of the world — a thumbs-up means "very good" in Hong Kong but can be an insult elsewhere. The three noticed how regional sign language is, and that was where the idea came from: a glove that could help a deaf or mute person talk to a foreigner. As the article noted, the sign-language apps already on the market are mostly teaching tools, and almost none of them translate.

Meaning inferred from bend and bearing

The glove has three main parts: flex sensors at the fingers, a three-axis gyroscope, and a Bluetooth transmitter. The flex sensors and the gyroscope collect the data of the hand shape and its movement; the Bluetooth transmitter sends that data to a phone app. The app is the processing centre — it matches the sign against the phrases in its database and then outputs text and speech. All the main components sit on the back of the hand.

Jacky joked that the demonstration set includes "I love you", handy for confessing to a classmate you like. And when the team saw the bird in the UNWIRE logo, they set the sign for "bird" to translate as UNWIRE.HK.

Eighty-five per cent accurate

Noddy led the hardware. He said the hardest part of the build was tuning the sensitivity of the sensors: there are so many possible hand positions, and the sensors are so sensitive, that the smallest change in a gesture changes the data. A lot of time went into the maths of how that data was calculated, to make it more accurate. After the tuning, the team put the glove's accuracy at about eighty-five per cent, though the database did not yet hold many phrases.

Before making the glove, Noddy had considered several other ways of analysing the motion of a hand. They had found plenty of software online that could do it, but most of it needed a computer connection — not convenient enough, and not accurate enough — so in the end they built the glove and wrote the program themselves. In the app, the numbers at the bottom left of the screen are the raw data coming off the glove: the degree of bend in each finger and the bearing of the hand.

Output in several languages

Jacky handled the phone app. At the time of the interview it translated Hong Kong sign language into English, and he hoped to add French. Asked why those two languages and not Chinese, he explained with a laugh that the development tool they were using did not support Chinese, so there was no way to translate into it yet. Watching how widely the data swung, he brought in an open-source mathematical model to denoise it.

Off to the United States

The three took only about three months to build the prototype, and it won them a place at the International Sustainable World Energy, Engineering and Environment Project Olympiad, as well as the championship of the senior-secondary invention category at the Hong Kong Student Science Project Competition. All three said they wanted to keep refining the product, fill out the phrase database, and then put the glove in the hands of deaf and mute users to try. Entering a phrase in the app and signing it once is all it takes to log a new entry.

PC Market's business technology desk covered the glove as a hardware story: flex sensors and a gyroscope reading the hand, Bluetooth to a phone, a phrase database doing the matching, and speech and text coming out the other end — presented as a piece of wearable engineering rather than a school project.

On Friday at the George Brown Center, students from around the world and from across the United States showed the public their research in science, technology, engineering and mathematics. Chinese students from the mainland, Hong Kong, Taiwan and Macau presented their work too — among them three high-school students from Hong Kong who had developed a sign-language translation glove to help deaf and mute people.

The three students from Sun Kei Secondary School, Hong Kong: "My hands write what my heart says."

Noddy Chung, Sun Kei Secondary School: "There's speech output as well. While the signer is signing, the phone can be shut, it can be left somewhere else, and the speech still comes out — so the signer can talk with other people."

Michael Ng, Sun Kei Secondary School: "Put the glove on, make a sign, and the meaning appears on the phone."

They hope the product will find real use.

Jacky Chan, Sun Kei Secondary School: "Not just to compete here, but to be used by hearing-impaired people in the future."

The report went on to a Form 11 student from Jilin who electrolyses water cheaply and efficiently to produce hydrogen, and noted that the projects on show had been selected from more than four hundred and twenty entries, with students from more than sixty countries taking part. The chair of the Texas Workforce Commission, Andres Alcantar, came to see the work: "They provide solutions for the real world, and the contribution of those solutions reminds us why we support these outstanding students."

Reported by Yu Qing and Hong Yu, NTD, Houston, Texas.

Hong Kong secondary students are endlessly creative. Christian Alliance Sun Kei Secondary School has made its name abroad with a sign-language translation glove, taking the bronze award in the engineering category of the International Sustainable World Energy, Engineering and Environment Project Olympiad (I-SWEEEP). It shows that the standard of Hong Kong students' research can meet the world's, and that their work can contribute to improving the human environment.

The 2017 Hong Kong Student Science Project Competition — organised by the Hong Kong Federation of Youth Groups, the Education Bureau and the Hong Kong Science Museum, with the support of the Innovation and Technology Commission and the Hong Kong Science and Technology Parks Corporation — sent three teams to Houston from 3 to 8 May to compete at I-SWEEEP against more than four hundred entries from close to seventy countries.

Four competition categories

The Hong Kong teams did well this year, with Sun Kei's sign-language translation glove taking bronze in engineering. The glove translates sign language directly into speech and text, shown instantly on a smartphone, removing the barrier between deaf and mute people and everyone else.

The two other Hong Kong entries both came from Carmel Pak U Secondary School: "Making the Most of Rain", which turns rainwater into energy so that natural resources are used well and dependence on fossil fuels reduced; and a study of using green algae to convert coal into hydrogen, with hydrogen as the environmentally friendly fuel.

I-SWEEEP 2017 divided its entries into four categories — energy; engineering; environment (management and pollution); and environment (health and disease prevention) — each with a grand champion, gold, silver, bronze and merit awards. The three Hong Kong teams had been selected at the preliminary round and exhibition of the 2017 Hong Kong Student Science Project Competition, held on 18 March.

The Hong Kong Economic Times reported the Houston result, and framed it the way the business press tends to: a group of Form 5 students had built a wearable that solved a communication problem, and had been judged against four hundred entries from close to seventy countries to do it. The piece also ran on the paper's TOPick channel.

The world has translation books and dictionaries in every language to help us talk to people from other countries, while deaf and mute people can only express their thoughts through sign language. And if they meet a hearing person who does not sign, who is there to translate for them? A group of senior secondary students put their wits to work and invented a sign-language translation glove — which not only won the senior-secondary championship of the 2017 Hong Kong Student Science Project Competition, but stands to benefit society.

Translating sign language, breaking the barrier

The idea began with something funny. Jacky said: "One time the vice-principal put out a hand towards Noddy, meaning to ask him for a pen — and Noddy thought he was gesturing for a handshake, so he put his hand out too. That comic moment gave us the idea: hands can carry meaning just as speech does, and yet society has no tool for translating sign language."

As it happened, several of them had known a little sign language since they were young. Noddy said he had learned sign-language songs in primary school and understood the difficulty deaf and mute people have in communicating: "Sign language is not the same from place to place — one gesture can have dozens of readings. We hope that signers anywhere in the world can speak freely through the app we invented, and break down the wall between them and society."

Calibrating the hand to cut the error

The students demonstrated the glove themselves: "First put on the glove, already paired with the app, then make the sign. The hand movement is scanned, the data is sent to the smartphone, and the meaning is spoken aloud and shown as text on the screen."

The work was not without setbacks, above all translations that came out wrong or failed to appear at all. Noddy explained: "One sign can be made in different ways — pointing forward means 'you', and pointing to the side also means 'you' — so a single word needs a great deal of data recorded at different degrees of finger bend. And everyone's palm size, finger length and hand shape are different, so the readings the flex sensors collect are constantly at odds."

To normalise the data between users, the students divided the bend of a finger into four levels. Before translating, the user presses a "calibrate" button, which records the maximum and minimum values of their hand so that the data can be applied to them — bringing the error rate down.

To America, for the international contest

Michael said: "Our database currently holds somewhere between twenty and forty words. To make communication easier in future, we'll enter data for many more words — we want genuinely barrier-free communication in sign language. On languages, besides the English we have now, we'd like to add French or other versions; in the long run the glove wouldn't only be for talking to local people, but to foreigners too."

On the strength of that innovation, the work took the bronze award in the engineering category in the United States. The students spent two months inventing the glove, and the whole design and build was their own work. They plan to add more vocabulary and more recognised languages, to widen how broadly it can be used.

The South China Morning Post's Young Post covered the glove for its student readership — the only article in the file written in English, and so the only one that needed no translation.

At InnoTech Expo 2017, held from 24 September to 2 October at the Convention and Exhibition Centre in Wan Chai, you can see as many as fifteen outstanding local research results up close and all in one place. The nine-day event takes "Learning from the past, creating the future" as its theme, and divides the hall into five zones — Sky, Communication, Sea, the Chinese Academy of Sciences interactive zone, and Pride of Hong Kong.

As a Hong Konger, I naturally looked at Pride of Hong Kong first. A good many of the works in that zone are highly practical, made for the benefit of people. Take the sign-language translation glove — this secondary-school invention helps break through the communication barrier for deaf and mute people. What matters most about the glove is that it is studded with sensors: when a deaf or mute person signs, the resistance of the sensors changes as the fingers bend and straighten. That data travels to a phone over Bluetooth, the phone app translates it, and in a matter of seconds the words the sign is meant to express appear on the screen — making it easy to talk to anyone.

Wen Wei Po's innovation desk previewed the nine-day expo at the Hong Kong Convention and Exhibition Centre, listing the exhibits visitors should look for in the Hong Kong Pavilion. The sign-language translation glove was named among them, alongside the university and institute research also on display.

Ming Pao reported the opening of InnoTech Expo 2017 and walked the Hong Kong Pavilion, where the sign-language translation glove was one of the exhibits the paper singled out: the wearer signs, the sensors read the hand, and the phone speaks and writes the meaning — a piece of student engineering set among university research on the same floor.

Headline Daily's live desk covered the opening: around a hundred and thirty exhibits over nine days, the world's largest walk-through kaleidoscope among them, and a Hong Kong Pavilion of twenty local research results including the sign-language translation glove — which lets a deaf or mute person be understood without an interpreter by signing into sensors on a glove.

The Hong Kong Economic Journal's startup channel covered the expo as a story about raising a generation on technology, running through the exhibits a family could see across the nine days — VR rides, science demonstrations, ten local universities and research institutes — and closing on the student work in the Hong Kong Pavilion.

"Through the sign-language translation glove developed by secondary-school students, deaf and mute people can have the meaning of their hand movements read out by the sensors on the glove, helping them to express themselves fluently."

A mainland syndication of the glove story, republished in April 2018 — a year after the Houston result — showing the project still circulating in the technology pages. The publication's archive cannot be read from outside its own network, so this entry keeps the citation only.