Cracking the last code: BYPPCVACMZQVYAP

Image result

In my last podcast episode, my quick-and-dirty algorithm for cracking a bunch of substitution cyphers made good work of all the encoded messages it was given - all except for one.

This last message goes as follows:BYPPCVACMZQVYAP

I tried tackling it by hand to no avail - with the repeated letters in the 3rd and 4th position, I could guess that the first word is "HELLO" or something like that. But I couldn't make it fit with the rest of the message!

It's very difficult to crack substitution cyphers on small messages because they contain less information, and several different substitutions of letters can make sense.

For example, consider this four-letter substitution cypher:TBBO

The only information we have about this word is that it has 3 unique letters, with the two letters in the middle repeating. It could be LOOK, BOOK, BEEP, BOOM, ROOM, or any such word like that. In this case, there's no hope in pinning down the message directly, unless we could gain more information about the word in question.

For the 15 letter message above, I think there is hope. I'm going to try a couple of ways to get ahead of it, and I'll post this to the github for my project.

1) Improve the language model.

I noticed that for short messages, it can find other messages with the same substitution pattern with lower entropy. In other words, the language model doesn't necessarily pick out the best one due to the simplicity of our language models compared to real natural language. This isn't as much of a problem with larger texts because it has so much more information to go on in separating out "real" from "fake" English.

My first thought is to move from bag-of-words to a bigram model, and work out from there.

I should also keep track of several top messages, to make sure that the real message doesn't wind up in space 2 or 3.

2) Count every possibility.

In the other cases, it wasn't possible to check every possible letter-and-space substitution because that meant checking 27! possibilities. That's a big number!

10,888,869,450,418,352,160,768,000,000

That's around 10^28. I said on the show that it has "28 zeros" as a shorthand for this order of magnitude, but of course I realized that I misspoke since it's actually not 28 zeros but 28 digits. I think most people got the point!

Anyway, in the case above, only 9 unique letters are used: ABCMPQVYZ. This means that there are far fewer substitutions to check. After selecting on of the letters as the space, there are only 8 substitutions to make! Let's do the math on that:

27! / (27-8)! = 27! / 19! = 27 * 26 * 25 * 24 * 23 * 22 * 21 * 20= 89,513,424,000

At around 89 billion possibilities that no small iteration even for a computer. And no way I'm going spin up an Amazon cluster for this thing. BUT! This can be whittled down further with some educated guesses about which letters could arise, making exhaustive search reasonable.

If you crack this one before me, either email the show localmaxradio@gmail.com, or message me on Twitter. I'll give you a shout out in the next episode!

Impress your Friends by Finding the Best Places

Designing the algorithm for Foursquare’s venue ratings is one of the best things I’ve worked on in my career. I hear people tell me that if they want to go to a good place, they make a cutoff on our 1-10 scale, say 8.5, and limit their choices to the select few elite places.

To me that sounds a little strict, but the fact remains that the Foursquare venue ratings are a great way to tell the difference between a good spot and a bad spot, and to assess the overall quality of a restaurant or bar before you go. Stephanie Yang and I spent a lot of time ensuring that our ratings are the best in the business, and I’d put these up against any venue rating system out there in terms of quality and accuracy.

Have you ever wondered how we do it? Well, we don’t give away all the secrets, but Stephanie and I wrote a blog post for the Foursquare Engineering blog called Finding the Perfect 10 where we break down some of the methods we use around venue ratings.

Watch this Space in 2018

As the year wraps up, I just wanted to let you all know that I am planning a really exciting new project for 2018 that is going to take my "content-creation" to the next level. If you like to hear about probability, travel, technology, and all the topics I discuss - this will be something to look forward to!

I know that's cryptic - but I wanted to tease it out before the new year. In addition, I've taken 2 really fascinating trips this year that I wanted to blog about. The first was my trip to Kentucky and Tennessee to see the eclipse, and the other was my trip to the Fort Mojave Indian Reservation to do some volunteer work in that community. I was heavily involved in planning the latter which meant that this blog was a bit neglected, but I hope to give a full report on it soon!

I have a whole bunch of posts lined up too - and at Foursquare I've been shopping some talks with the "Data" crew that I may want to share publicly.

But for tonight - I'm going to go out, celebrate, and possibly have my yearly slice at Koronet Pizza. Judging from the jackets, it clearly wasn't as cold 3 years ago as it was tonight.Happy New Year everyone!

 
 

I actually witnessed SQL Injection

SQL injection is one of those hacks you can do on websites with really bad security practices. It can occur whenever your database query includes user input. If the user puts something you don't expect, they can alter the database in ways that you don't expect.

A funny example - which is kind of famous in engineering circles - is given in the webcomic XKCD.

Now about 10 years ago, I coded up a site called Stickymap. It was a local search where users can post locations in their neighborhood that are interesting and leave description. It was coded in PHP. You can secure PHP if you're careful but it's very difficult to do so. If you use PHP in your organization, there should be very specific rules around running SQL queries.

biglogo

biglogo

Well - one of my queries did not escape the user generated data. And, long story short someone changed every single venue name to "Bureau Veritas". Every single one. In the world.

After I investigated, I don't think that this was the intent. I think that the user was trying to add a (very spammy) description to a single venue that short-circuited the query so that the "WHERE clause" didn't make it in. For those of you who don't know, the WHERE clause in an UPDATE statement tells the database which items to update. If there is no WHERE clause, it'll update everything. Pretty insane, right?! It should probably update nothing.

I wonder how that person/spammer felt after they did this. Where they shocked? Did they move on to another site? Who knows!?

Fortunately, I had enough backup data to restore the Stickymap database while I was in San Francisco. Of course this always happens when I'm in San Francisco away from my home computer!!

Furthermore, I plugged up the security hole on the site. It's pretty cool that the security hole was left unexploited for 10 years and then all of a sudden was found. Who knows what problems we have lurking in our more critical systems? I like to hope those are more widely tested. You also want to see systems that hackers are constantly trying to exploit because that means that the owners of that system have been forced to plug the security holes. For example, I would rather trust software that's been cracked and plugged a few times in the past than software that's never been hacked but also never left out in the wild either.

Anyway - if someone out there wants to tell me there are more security holes in my site - let me know! But please try not to destroy Stickymap - it's my fun mid-2000s space on the internet and a reminder of how far we've come on local search.

And if you are the accidental culprit and you come forward, I'll either interview you for the blog, or I'll owe you a beer!

Talkabot: The bot conference in Austin

 
IMG_9305.jpg
 

Last month, I also attended the Talkabot Conference in Austin, TX. I gave a 30 minute presentation on Marsbot. I guess you could say I am on a national tour! This time, I shifted focus to how we're adopting everything we've learned about user context (location stops, taste likes, time) to send messages that are really useful.

The conference was great - it was actually my first time in Austin. I had a very warm reception from the team at Howdy and on the last day got to go out on a trek to the salt lick for some BBQ with the founders of Kip, and reps from Slack. At the conference itself, there was a lot of talk about developing standards for chat bots, and building tools and platforms upon which these bots will be built. I loved ordering coffee from a bot barista on kik who in addition to giving you coffee also pitches you his screenplays.

 
img_9338

img_9338

 

Even though chat bots have existed for a while, there is a sense that we've hit an inflection point and some of the killer apps are coming. There is hope that chat could be the next great platform for innovation.

I've been asked to share my slides. Here they are: talkabot-marsbot-presentation-1

If that wasn't fun enough, Marsbot actually attended the Foursquare Halloween party last night!

Marsbot Slides for Industry Talk at RecSys 2016

I recently attended the 2016 conference on Recommender Systems at MIT with my Foursquare coworkers Stephanie Yang and Enrique Cruz. We had several contributions - 2 posters and a 20 minute industry talk on Marsbot.

Marsbot is a character in your pocket that acts as a text-based service for local recommendations. I've been working on it for a while, and we were able to do a full launch a couple months ago. I have so much to say about this project and I hope to expand on it more on this blog soon!

For now, a bunch of people have asked me to post slides from my talk at RecSys so I will post them here. I hope the video of the talk become available soon.PDF of the slides:recsys-marsbot-presentation

Swarm Creator Coins: Another Hackday Project Launch

 
IMG_8904.png
 

This post is about another hackday project on Foursquare that we were able to quietly launch into the Swarm app recently.

I always felt that Foursquare and Swarm should give recognition to people who contribute to our database. We want our venue database to be as accurate and up-to-date as possible. I also feel a little bit of pride whenever I’m the first person to add the venue to the database.

For example, I added the 7-Eleven in the East Village to the Foursquare database on it’s grand opening, and now it has over 150 unique visitors on Swarm! Now that’s not bad, but there are some power superusers who have created things that are a lot bigger. You need to be first, and it needs to be someplace important, so it’s tough.

What creator coins does is reward users who create great venues. You don’t get points just for adding any old thing to the venue database – but if you add something that stands the test of time and becomes popular, you will be rewarded with coins whenever you check in.

Also when your friends or friends-of-friends check in, they will recieve some coins, and they’ll be reminded that you are the creator. In that way, you get some recognition!

So, I built this with some pointers from the Swarm team during our hack day a few months ago. We had to get some of the copy I wrote translated, but it’s not live on the site. Tweet at me if you find a good one! Here are some examples:

Typical example, street car with 90 checkins:

https://twitter.com/bobspecht/status/768096429982134272

This person created an airport terminal venue with almost 34K checkins!

Here’s my own creation of Atlantic Center. I created this venue a few months ago when I was reorganizing Foursquare Supervenues. Turns out that Atlantic Terminal Mall and Atlantic Center are 2 different things, even though they have the same owners and are connected. Confusing!

 
IMG_8902

IMG_8902

 

And here are some cases in my own feed where I learn about Friends of Friends who created places that I love going to:

 
IMG_8903

IMG_8903

 

YASC trip to Ghana - Part 3

Now in Yamaransa, I was on the business consulting team which is similar to the work I did in Trohilo, Nicaragua in 2013 and 2014. We set up a makeshift consulting firm in a couple of classrooms with half-built desks and lots of flies. Men and women who run local businesses come in periodically. We get to know them personally, and learn about what they do. Then we analyze their business model and come up with some recommendations.

Since the YASC group has been there before several times had been there before several times, several of the entrepreneurs had been through this before. In some cases, you get reports of incremental improvements based on changes decided upon in the previous year, which is great. But of course, the YASC volunteers can’t come up with a good plan for every group and just like back home not every plan pans out the way you’d hoped. But essentially, we try to make the most of the time we have with these people by getting to know each other, learning about their business and their goals for the future, and brainstorming ideas.

For example, one man that I met with was a tailor named Moses at the same time my fellow volunteers Faith and LaShawn met with the dressmakers next store. Now Moses told us that he had met with the volunteers last time and not much had come out of it to help his business directly – but that he had started an apprentice program. [I remember being told in translation that he said he was inspired by the help from YASC last year to start this program, but that almost seems to good to be true, so I’m going to put that as a “maybe”]. He asked if it was okay if he brings his apprentices to the meeting, and I said “sure”!

Now I had assumed that he’d be bringing three or four people – but I was surprised when about 20 people showed up! We got the impression that  the dressmakers would all men women and the tailors would be men, and that was true but the apprentices were mixed in gender.

One of the things we went through successfully was pricing. Although I wish I had more time, I went through several different examples of products that could be produced by the tailors. I tend to look at three main variables: the total cost of the input materials, the price that is set for the item, and the time it takes to put it together. Similar to what we found in Nicaragua, the price often reflected the input materials but didn’t take into account labor as well. For example, the back-of-the-napkin calculation we did for a quick 1-hour shirt was 16 cedis of profit per hour, which the 3-day suit was about 3. I’d expect it to go down because the suit is guaranteed work for 3 days, but that differential seemed extreme. Moses agreed. I hope one of those 20 apprentices likes to calculate this stuff, because then maybe they can make a table for all the products for everyone!

Now while that “profit per hour” calculation is helpful, there’s definitely more I would do if I had more time. The pricing strategy once you get that information is important, because you still need to take into account how much work you’re getting (customer acquisition costs) and whether the time spent on a low-profit item is really displacing time that would be spent on a high profit one. But we needed to give the tailors and the dressmakers some time to actually sell to the group – because that generates a one-time infusion of significant business for them and I got a really cool African shirt!

There were several problems that seems to exist in common across all of the entrepreneurs, and many of these same problems existed in Nicaragua as well. First of all, in a community like this you have many customers who expect to be given goods on credit, and they end up not being able to pay. This is not such an easy problem to deal with – because if you start saying “no” to products without payment, you’re going to get pushback from some of these people, who ultimately may be family or close friends.

Another issue that comes up is how to build savings. Given the tiny amounts that people are making, it’s very difficult to put something away. Once they take care of their immediate family, people are often expected to help extended family in need as well.

In addition to my work on the business team, I was also able to see what the education team, and the group working on Yamoransa’s new ICT center was up to. I was able to sneak in a couple of math classes on Friday (August 12th), and because I needed to come up with something at the last second, I decided to talk about the handshake problem. It seemed to go pretty well – fortunately we had very exceptional translations from GhanaThink volunteers as well as from AFS Ghana throughout the whole process making this possible. I changed the handshake problem to the fistbump problem, and I had the students (age maybe 9 or 10?) get up in groups of various sizes to see how many fist bumps they need so that everyone in the group matches with everyone else. The result is a combination of fun interactive trials plus an exploration into a variety of problem solving techniques. It also introduces some ideas in computer science which I really love.

While the students didn’t seem to have enough access to mathematics enrichment, they don’t seem to have the same aversion to math that you would find in classrooms and among the general population of the United States. In several conversations, math seemed to be a subject that the people enjoyed and wanted to learn more about.

And finally – I got to organize books for the library in the newly minted ITC center. Because internet access is limited, this is being set up with old-school paper records. Some of the books looked interested since many were children’s books on a variety of scientific topics like space. Some books contained information about far away places such as Angola, or New York. I had to read the book about the latter – which seemed to take an overly positive view of Tammany Hall, and also had a section on Donald Trump as a real estate developer and reality star from 2010. I wanted to write stuff in the margins, but I stopped myself.

This new center is really remarkable – the rooms looks open and comfortable, and part of the purpose of this center is to provide computer education and literacy. Now their internet access availability is going to be very limited, but there’s still a lot that can be done. For example, thousands of books can be uploaded into e-readers. I’m really excited about the idea of having an offline version of the Khan academy available. If that’s the case, you can have a whole generation of students and teachers who have access to materials up to the college level. And I have no doubt that people will want to use it. If that’s the case, what effect does this have several years down the road? Seems like it will be very positive, but time will tell.


Finally, I know I haven’t mentioned all the great people I worked with. There are too many to list! But on the business development team, we had Lisa Unsworth leading it, and I worked closely with LaShawn Warren (we had some very interesting discussions with the breadmakers), Anke Tietz and Faith Lin (who were great with the hairdressers and dressmakers), Aric Sangruchi, Sam Blango, and Nick Mason. Also a shout out to Hamilton Barnes volunteering from GhanaThink was there the whole time translating some very complex stories back and forth, particularly with the breadmakers!

IMG_8426.jpg
IMG_8682

IMG_8682

IMG_8569

IMG_8569

IMG_8831

IMG_8831

IMG_8496

IMG_8496

IMG_8426.jpg

IMG_8426

YASC trip to Ghana, Part 2

Now on to Yamaransa, the community where we were working last week. Yamaransa is located between Accra and Cape Coast, 2 important cities and Ghana. Yamaransa lies at an important 3 way intersection: to the west is Cape Coast, to the East is Accra, and inland to the north is the major city of Kumasi in the Ashanti region.

YASC along with several other organizations have been working there since 2011. The focus has been on education, health, business, and the construction (and sustainability) of a community center. The priorities have been set by the community leaders themselves, not outsiders. The idea for us is to engage in some cultural exchange, and hopefully make the community better off and help them achieve their goals – be it education, economic development, or anything else. Because this has been an ongoing effort, we’ve been able to measure our efforts and learn from our mistakes. It was great to see some of the progress that has already been made, and I hope to see progress continue for years to come.

When we first got to the site, we were treated to a host family visit followed by a cultural fair put on by AFS Ghana – one of our ground partners. My host was a catholic priest in town. His sister was also visiting – a woman in her 20s who was finishing up high school. It was clear that it was not due to monetary constraints and not to effort. I asked her what her favorite subject was, and when she said math I said “ah ha! We need to talk about this!” because it was very difficult to find common ground otherwise. I recited the proof that there are an infinite number of primes, and she wrote down a multiplication table for modulo arithmetic. She said “I wish you could come by and teach math every day!”

At the cultural fair I saw a lot of things that were familiar. At the booth of Ghanian games, they had one that was very similar to Mancala. All the different tribes are still confusing to me – the people in Yamoransa speak Fante (and that’s in the central region), and the Ashanti to the north is it’s own region. They have their own kings (local leaders) independent of the government.

In religion, they combine western religions with traditional African beliefs. There are so many churches and Mosques all over Ghana – probably the majority of non-house buildings. In the town there are Catholics, Protestants, and even the Mormon church has a significant presence. And there is also a large Muslim population, but not too many Jews. Ghana doesn’t seem to have the religious strife that would be expected from such a combination (and that happens in other parts of the world). They have developed a mature and tolerant attitude towards the different religions and there is no evidence that any group preaches hate or distrust towards any other group.

Interestingly, I identified more with some of their traditional African traditions than their western religious ones – and in some cases seemed more familiar to me. They have very specific ceremonies for different events in life such as birth, adulthood, marriage, and death. The baby naming ceremony takes place on the eighth day after birth. The people there can have both western names, as well as 14 possible day names based on gender and weekday of birth. The ceremony involves both water and wine – and I wasn’t sure on the exact symbolism there but maybe it’s two different aspects of life.

The funerals are very interesting. If the person was old – the funeral becomes a celebration of the person’s life. The coffin could be designed to represent them (perhaps their profession) – and they make giant posters that you see all over town describing relatives who have passed on. The funeral looks more like a send off to the world to come.

Also, we were able to hear the traditional horn that is played in honor of the Ashanti kings. It looks and sounds exactly the same as the Shofar that is heard in synagogue every year in Rosh Hashana. And finally – we were treated to lots and lot of drumming. Drumming for music. Drumming for communication. Drumming for ceremonial celebrations. Like pretty much constantly the whole time.

IMG_8342-1.jpg
IMG_8342-1.jpg

IMG_8342

[ Up Next: The Volunteer work ]

YASC Trip to Ghana - Part 1 of 3

I just returned from Ghana on a volunteer trip with the Yale Alumni Service Corps, and what an amazing trip it was! This was my first time visiting Ghana, or any country in Africa for that matter. Turns out that I didn’t know a whole lot about the country and I had a lot to learn. I’m going to provide here an outsider’s perspective based on a 2 week trip so take my words not as encyclopedic knowledge but as the initial perception of one American which subject to change.

I arrived at the Airport in Accra on July 29th for a week of tours which included Togo and Benin. Those were countries I knew even less about (!) but let’s stick to Ghana.

Ghana gained its independence from Britain in 1957, and that event is seen as heralding the decolonization of Africa from the Europeans. The United States sent a delegation including a couple of household names today: then-Vice President Nixon, and Martin Luther King. At the time Ghana was known as “Gold Coast” – following the same naming scheme as its neighboring “Ivory Coast” which tell us what explorers and traders were initially doing there. But in order to symbolize their independence, the founders decided to name their country after the ancient Ghana Empire in west Africa. In this way, the idea was to take an ancient, homegrown civilization, and reimagine it as a modern democracy.

In fact, some of Ghana’s democratic institutions are similar to what we have in the US. In foreign policy, they were part of the non-aligned movement (neither with the US or the USSR in the cold war). Domestically, we Americans may be familiar with some of the government functions. But while Ghana was never a communist state, the founders instituted socialist-style plans for industrialization. If you look around today, you can’t say that’s been too successful. Corruption by government employees is seen as an impediment to economic growth. Heavy industry would have been better off under private initiative, if only a consistent regime of laws and rights can be relied upon.

After several coups and periods of instability, Ghana today has a stable democracy with competitive elections and a functioning judicial and legislative branch. That doesn’t mean that the government works for the people as much as they would like – but change is possible through the ballot box as opposed to violence.

So enough about that – what about the regular folks? When I was walking and riding around in the bus in Accra, I had never seen so many people working and hustling so hard. If you watch the people on the street, you can see that everyone is on a mission – and I’d say this is even more than in New York City. Most people are trying to sell things. Many are carrying large items, moving them from one place to another. Some people are dressed nicely walking briskly to meetings or events. You can see still other people with books and backpacks on their way to school. Hard work and initiative come naturally to Ghanaians, and I’d say they are a nation of entrepreneurs.

In central Accra there are lots of large construction projects. Like in Downtown Brooklyn, they seem to be on a bit of a building spree. But once you get outside the city, you notice something very interesting: a lot of half-built houses made of cinderblock.

When we in the US want to save for a house, we might open up a savings account, or perhaps invest in the money market and mutual funds. Then, we you have enough for a down payment, we can get a mortgage, and voila – we’re in the house! When these Ghanaians are saving for a house – they literally see a partial house! They purchase each cinder block when they can in the hopes that they can one day finish. This is the best option for many people, but it comes with many risks. Do they have the title to their land? The government can come in and give it to a third party in the name of economic development. Are they protected from natural disaster or theft? I doubt the insurance market is well developed. Can they sell half a house? I could imagine having a good market for that, but it appears that it probably isn’t.

Still, our savings and mortgage strategy is also fraught with risk and it’s a matter of mitigating those risks and improving efficiency. There might be economic opportunities in Ghana for solving problems of land-title, insurance, real estate markets, and division of labor for home builders.

Buying things in West Africa is also too much for me. I’m not used to bargaining on the spot! It’s stressful. Apparently I was very good at it when I negotiated for a hat in an Accra market. I have a hat already, why do I need yours? Yeah, I understand this is an African hat, but my nerdy tourist hat perfectly fits my needs. I don’t know about that design. So I got them to come way down on the price. But then somehow I didn’t leave the shop without buying a Ghana Soccer Jersey (awesome!) and a watercolor painting (I don’t need it!). I came home with a lot of cool stuff – but I would have looked around more if I could eye a product without the assumption that I was going to buy it.

While I found the vendors to be overly aggressive, I also found them to have a certain straightforwardness. Sure – they all claim to have a “good price just for you” and to be your friend – but I didn’t catch anyone trying to mislead about their products. And in a few cases they answers my questions correctly even if they knew it wasn’t the answer I wanted to hear. I bet with a little practice one could understand their “lingo” and be able to make wise purchases.

IMG_8342-1.jpg
IMG_8342

IMG_8342

[ Up Next: Arrival in Yamaransa ]

Another Quora Math Answer about Split-Complex Numbers

Well, I’ve been on a role with Quora math answers recently! One person asked the question of whether you can have a negative absolute value. In other words, could you have a negative distance between two points! Rather than dwelling on all the rules this would break (who needs rules!?) I decided to construct such a system.

And I found that it looked like something I’ve seen before: the split complex numbers. The one application I’ve seen for these numbers is in an online dating app, as presented at a RecSys workshop in 2012!

In this number system, you have a new special number called “j” which lives outside our usual number system. This number has the special property that j * j = 1.

The dating application works like this:
You have people of the same gender who are similar (positive numbers)
People of the same gender who are different (negative numbers)
People of the opposite sex who are good matches (positive j numbers)
People of the opposite sex who are bad matches (negative j numbers)

These assumptions correspond to mathematical statements (capital letters for people of the same gender, and lowercase for people of the opposite).
If you’re similar A and A is similar to B, you’re similar to B(1*1=1)
If you’re different from A and A is similar to B, then you’re different from B (1)(-1) = -1
If you match a, and a also matches B, then you’re similar to B (j*j = 1)
If you don’t match a, and a matches B, then you’re different from B (-j * j = -1)
And so on!

So here’s someone talking about negative space and mathematical impossibilities, and we end up with an online dating application! Yeah, I realize this is a heteronormative number system* that also reduces human personality to a single dimension – but still it’s pretty cool!

*That must be why it wasn’t part of the Yale curriculum.

Here’s my full answer on Quora
Here’s the RecSys workshop paper I referenced

News Corporation Sells Amplify

I worked at Wireless Generation early in my career. It was the education company that was bought by News Corp in 2010 and became Amplify.

You can find some articles on it here and here, but the short story is that after 5 years at News Corp, the company wasn’t performing as well as they had hoped, and they sold it to private investors.   There were also massive layoffs.

Wireless Generation/Amplify is a data driven education company. A large part of the focus when I was there was helping teaching in the early elementary school years ensure that all of their students learn to read and learn basic math skills. It goes without saying that getting this right for kids is really important, and in the 2000s we were starting to see internet-scale data on this for the first time. Sometimes at Foursquare, we’ll improve some click-through-rate by 1%. But if you improve reading-rate by 1%, think about how many lives are changed for the better!

Wireless Generation also built some of the first open-source curriculum for the internet, and it looks like Amplify now has a really fun math game. The positive side of my experience there, along with some talented employees, was around the products we were building.  The downside was that despite rhetoric to the contrary, management style was much too top-down for my liking. Working on a large contract for the NYC DOE was particularly painful since decisions were made by government bureaucrats and were sometimes politically driven. I can probably write several posts on frustrating times at Wireless Generation!

I can only speculate on what went wrong, and even if the articles had gone into more detail I would be certain that the story from the inside is completely different. What is it that News Corp miscalculated? As far as I can tell, there weren’t any major setbacks.

It turns out that while the original sale was going on, I was taking a course on business strategy at NYU. I had emailed my thoughts to the professor, and I was able to find them. This is an excerpt from December 2nd, 2012, printed as is (along with some awkward phrasing!)

A lot of people are asking why news corp want to get into education.  Clearly, the newspaper business is not a great one to be in right now – maybe they feel like they need to do something.  I feel that they might might want the company because they want expertise in digital content distribution (I developed hand-held and web applications while I was there).  However, they’d probably be better off just hiring a much smaller team to do that.  Someone said this is just about Rupert Murdoch trying to build a legacy.  I don’t know.

Other people are asking whether it will work.  They’re keeping the same management team.  I have a feeling that WirelessGeneration‘s growth is now going to be heavily subsidized by news corp.  But after Monday’s class discussion I’m wondering what news’ other businesses will get out of it.

Of course, I still didn’t mind getting a check for shares.

Because their products are so important, I hope that Amplify can refocus in the future. I can see a few things going for it:
– It will be under the leadership of the original founder, Larry Berger. He is a capable leader, knows what he’s doing in education, and I’m sure he’ll have big plans.
– It will have a smaller, more focused team. If it has 400, it’ll be the same amount as when I was there.
– The ownership will be private. There will be no parent corporation in an entirely different industry trying to steer the agenda.
– If they get to keep their amazing office space in DUMBO.. can’t beat that!
This is on the patio – taken by me in 2007.

It sounds like they are going to try to refocus on the original mission, this time with a much more experienced team. I’m feeling a lot more optimistic for them then I was 5 years ago – this could be the low in a turning point of sorts!

Also – I know this could be a difficult time for people who are still working there, so I wish you all the best. If you are a current employee or recently laid off and you need a new job, I’d be happy to meet up and show you a demo of what we’re working on at Foursquare. I have a lot of respect for anyone who is working through all the issues at Amplify, particularly the engineering and product teams.

Of course, I still didn't mind getting a check for shares.Because their products are so important, I hope that Amplify can refocus in the future. I can see a few things going for it:- It will be under the leadership of the original founder, Larry Berger. He is a capable leader, knows what he's doing in education, and I'm sure he'll have big plans.- It will have a smaller, more focused team. If it has 400, it'll be the same amount as when I was there.- The ownership will be private. There will be no parent corporation in an entirely different industry trying to steer the agenda.- If they get to keep their amazing office space in DUMBO.. can't beat that!This is on the patio - taken by me in 2007.It sounds like they are going to try to refocus on the original mission, this time with a much more experienced team. I'm feeling a lot more optimistic for them then I was 5 years ago - this could be the low in a turning point of sorts!Also - I know this could be a difficult time for people who are still working there, so I wish you all the best. If you are a current employee or recently laid off and you need a new job, I'd be happy to meet up and show you a demo of what we're working on at Foursquare. I have a lot of respect for anyone who is working through all the issues at Amplify, particularly the engineering and product teams. 

Appearance on BK Live for Beyond Coding

Over the past few months, I've been involved with a program called beyondcoding.io, which is a series of courses at New York City tech companies designed to develop career skills for people who are entering the tech industry.

I produced one of the classes with Maryam (see our NLP talk) at Foursquare on technical communication. I would describe it as a workshop that involved talks by Maryam and myself as well as some audience participation and group communication.

On Tuesday, I was interviewed on BK Live, which is a show on BRIC TV (a Brooklyn-focused television station in New York). Also appearing on the round table was Bethany Marzewki who works at stack overflow and did a great job organizing Beyond Coding with the New York Tech Talent Pipeline. This also features program students Keanna Hines and Shlomo Maghen!

Ok enough of the background, here's the full video!

A few notes on what I said:

  1. I really wished I played up Foursquare and Swarm more, especially for high school and college aged people. I mean - if that were available to me when I was in high school or even much younger, I'd be all over it! For one, I'd compare checkins to the (now defunct) North and South cafeteria in Weston High School. I was sort of vague on whether Foursquare or Swarm would be something that could get a teen interested in technology. Obviously there are certain applications, like do-my-taxes applications that isn't going to get anyone interested. I should have put Foursquare on Swarm squarely on the other side of that!

  2. I don't condone cheating.. seriously. Good thing I'm not running for office, or that'd be a great campaign ad for my opposition. I think modding games, though, and learning how to hack code in the process, is great. So long as you're not betting on the games.

Introducing Foursquare for the iPad, the best way to plan your holiday travels.

Introducing Foursquare for the iPad, the best way to plan your holiday travels.

Natural Language Processing at Foursquare

Last month, Maryam Aly and I gave a workshop for NYU tech week where we spoke about how Natural Language Processing is integrated into the Foursquare app and our technology stack. Later, we gave the students a hands-on introduction to the nltk toolkit.

Hakka Labs took video of the first part of the workshop and posted it. Here is it:

https://www.hakkalabs.co/articles/introduction-natural-language-processing

On this 4sqDay, we’re celebrating our amazing global community of superusers. Happy 4sqDay!

On this 4sqDay, we’re celebrating our amazing global community of superusers. Happy 4sqDay!

Digging into the Dirichlet Distribution

This is a link to my talk on the Dirichlet Distribution at the machine learning meetup:

http://www.hakkalabs.co/articles/the-dirichlet-distribution/

The open source project I’ve referenced lives here: https://github.com/maxsklar/BayesPy Feel free to jump in if you’re interested!  I have a paper on it that unfortunately did not get accepted to aistats (they cited lack of impact; I disagree).  I’ll try to fix it up and get it on arxiv in the next few weeks.