Kamusi: Eliminate Language Barriers
Tax deductible
I’ve been working on Kamusi for 21 years. The project started (6 years before Wikipedia) as a dictionary for the Swahili language. The idea was that a lot of people would pitch in to write the dictionary, and we would share it for free, with Swahili speakers in East Africa and anyone else who was interested.
It turned out that a lot of people were interested, not just among the 100 million people in the Swahili zone. Speakers of other African languages started asking, could we do the same for them? We said yes, but soon found out that a dictionary among several languages (multilingual) was a lot more complicated than a normal two-language (bilingual) dictionary.
As we explored how to put together a good multilingual dictionary, we found out that the reason none existed was because compiling one was impossible based on the usual bilingual methods. So, we developed an entirely new model: everything is based on concepts, not spellings. The result is a dictionary that treats each language separately (monolingual), with all of its own ideas and terms, and then joins ideas together across languages.
Every language gets its own free monolingual dictionary, and all those dictionaries link together so that you can translate among any language in the system. Once we started building the platform, we realized that we couldn’t just limit it to Africa. The final design can, in principle, handle all 7000 spoken languages.
In 2013 we went live with our prototype, and it worked like a charm with a small dataset for 20 languages. It was so nice, we were even recognized by the White House (with a plaque, not with funding) as part of its Big Data Initiative!
We learned a lot from the prototype, made some adjustments, added new features, and in 2015 began importing millions of terms from dozens of languages. We received letters of intent from more than 60 partners worldwide to join forces on innovative projects for their languages and technologies, using data that has never previously been assembled to do things for languages that have never before been possible. Things were getting cooler and cooler, and then, BANG, we fried the system.
What happened, in a nutshell, is that all the search engines (Google, Bing, Baidu, and a bunch you’ve never heard of) started crawling all of the drool-worthy new pages that showed the links among languages that had never before been web-searchable. We were getting so much robot traffic that response time for real people was taking more than a minute, totally unacceptable. We implemented some security patches to throttle back the search engines, and a process started going rogue – our database cache would swell up until our disk was completely full and we couldn’t serve any pages, we would purge the cache, and then we’d balloon until we were knocked offline again. So our server guy, working almost without pay, kept trying new patches until the hard drive actually failed physically.
After several weeks totally offline, our server host in South Africa installed new equipment and we reactivated the system in mid December. Hold the champagne, within hours the cache filled up and the new machine was offline.
There is a solution. We need to move from the “relational” database that worked for a relatively small amount of terms in a relatively few languages, to a “graph” database that can efficiently handle the billions of interconnections that we are assembling. The catch is that it involves money. Our lead programmer has worked many, many late nights pushing Kamusi forward, essentially without pay since we ran out of funding a couple of years ago. If we can pay him a remotely fair wage for his time, he can sink himself into coding the transition and getting us back online. Until then, we’re dead in the water.
If you knew how little I earn personally from Kamusi, you’d think I’m a nut to keep at it – for years, I’ve been working about 50 hours a week, and paid for 8 at the most. Nevertheless, I’ve told the programmer that I will take out a personal loan to get the project back afloat, if that’s the only way. I know how spectacular the underlying system is, and I know how it will change the game for people around the world once it is functional – think of how your life has changed with tools like Google and Wikipedia, and apply that idea to languages by the hundreds. While most people in the world don’t speak English, and won’t in their lifetimes, the data and tools we’re putting together at Kamusi could open up access to knowledge and technology for a great many people who have been excluded until now. But I can’t pitch that idea to funders if we don’t have a site to demonstrate, and I can’t demonstrate the site if I can’t pay for the programming fixes.
After the thousands of unpaid hours I’ve put into Kamusi, I know how close we are to delivering something truly revolutionary, and I know we can’t abandon ship here. What I don’t know is how far my credit cards can go to get us over this hump – I’m not a money guy, which explains a lot about how we got into this predicament. As soon as we implement the fix, we can get back to the real tasks, including importing and interlinking more than 10 million terms in about 50 languages with datasets at the ready, and introducing systems to bring in quality data for hundreds more. That’s why I’m asking you to help now, not for me but for the project, and for the many people who will be able to benefit from the knowledge and communication tools we can create going forward.
One more thing, Kamusi is a tax-deductible 501(c)3 non-profit in the US, and also a registered non-profit in Switzerland, so donations may qualify you for tax deductions in several European countries.
As well as chipping in until it hurts a little, thanks for sharing this appeal with others. It’s a moon-shot quest to build a free knowledge tool for every word in every language, and needs as much support as we can raise.
Thank you for your generous help!
It turned out that a lot of people were interested, not just among the 100 million people in the Swahili zone. Speakers of other African languages started asking, could we do the same for them? We said yes, but soon found out that a dictionary among several languages (multilingual) was a lot more complicated than a normal two-language (bilingual) dictionary.
As we explored how to put together a good multilingual dictionary, we found out that the reason none existed was because compiling one was impossible based on the usual bilingual methods. So, we developed an entirely new model: everything is based on concepts, not spellings. The result is a dictionary that treats each language separately (monolingual), with all of its own ideas and terms, and then joins ideas together across languages.
Every language gets its own free monolingual dictionary, and all those dictionaries link together so that you can translate among any language in the system. Once we started building the platform, we realized that we couldn’t just limit it to Africa. The final design can, in principle, handle all 7000 spoken languages.
In 2013 we went live with our prototype, and it worked like a charm with a small dataset for 20 languages. It was so nice, we were even recognized by the White House (with a plaque, not with funding) as part of its Big Data Initiative!
We learned a lot from the prototype, made some adjustments, added new features, and in 2015 began importing millions of terms from dozens of languages. We received letters of intent from more than 60 partners worldwide to join forces on innovative projects for their languages and technologies, using data that has never previously been assembled to do things for languages that have never before been possible. Things were getting cooler and cooler, and then, BANG, we fried the system.
What happened, in a nutshell, is that all the search engines (Google, Bing, Baidu, and a bunch you’ve never heard of) started crawling all of the drool-worthy new pages that showed the links among languages that had never before been web-searchable. We were getting so much robot traffic that response time for real people was taking more than a minute, totally unacceptable. We implemented some security patches to throttle back the search engines, and a process started going rogue – our database cache would swell up until our disk was completely full and we couldn’t serve any pages, we would purge the cache, and then we’d balloon until we were knocked offline again. So our server guy, working almost without pay, kept trying new patches until the hard drive actually failed physically.
After several weeks totally offline, our server host in South Africa installed new equipment and we reactivated the system in mid December. Hold the champagne, within hours the cache filled up and the new machine was offline.
There is a solution. We need to move from the “relational” database that worked for a relatively small amount of terms in a relatively few languages, to a “graph” database that can efficiently handle the billions of interconnections that we are assembling. The catch is that it involves money. Our lead programmer has worked many, many late nights pushing Kamusi forward, essentially without pay since we ran out of funding a couple of years ago. If we can pay him a remotely fair wage for his time, he can sink himself into coding the transition and getting us back online. Until then, we’re dead in the water.
If you knew how little I earn personally from Kamusi, you’d think I’m a nut to keep at it – for years, I’ve been working about 50 hours a week, and paid for 8 at the most. Nevertheless, I’ve told the programmer that I will take out a personal loan to get the project back afloat, if that’s the only way. I know how spectacular the underlying system is, and I know how it will change the game for people around the world once it is functional – think of how your life has changed with tools like Google and Wikipedia, and apply that idea to languages by the hundreds. While most people in the world don’t speak English, and won’t in their lifetimes, the data and tools we’re putting together at Kamusi could open up access to knowledge and technology for a great many people who have been excluded until now. But I can’t pitch that idea to funders if we don’t have a site to demonstrate, and I can’t demonstrate the site if I can’t pay for the programming fixes.
After the thousands of unpaid hours I’ve put into Kamusi, I know how close we are to delivering something truly revolutionary, and I know we can’t abandon ship here. What I don’t know is how far my credit cards can go to get us over this hump – I’m not a money guy, which explains a lot about how we got into this predicament. As soon as we implement the fix, we can get back to the real tasks, including importing and interlinking more than 10 million terms in about 50 languages with datasets at the ready, and introducing systems to bring in quality data for hundreds more. That’s why I’m asking you to help now, not for me but for the project, and for the many people who will be able to benefit from the knowledge and communication tools we can create going forward.
One more thing, Kamusi is a tax-deductible 501(c)3 non-profit in the US, and also a registered non-profit in Switzerland, so donations may qualify you for tax deductions in several European countries.
As well as chipping in until it hurts a little, thanks for sharing this appeal with others. It’s a moon-shot quest to build a free knowledge tool for every word in every language, and needs as much support as we can raise.
Thank you for your generous help!
Organizer
Martin Benjamin
Organizer
Brooklyn, NY
KAMUSI PROJECT USA
Registered nonprofit
Donations are typically 100% tax deductible in the US.
