Toward a New Great Compromise: How I would change the Constitution

Read Toward a New Great Compromise: A Constitutional Plan to Promote Liberty, Renew Democracy, and Restore Federalism

I recently completed a project where I decided to rewrite the US constitution. The changes were designed to fix several problems in the design of the senate and the administrative state. When I started thinking about how the design of the Senate is flawed, I found out that George Mason was saying some of the very same things back in the constitutional convention of 1787!

I’ve posted it in several places for public comment. I believe that the new executive council is designed to fill executive offices better than the current role of the Senate provides, and will provide a more harmonious and prosperous future. I’d like to know what you think!

Read Toward a New Great Compromise: A Constitutional Plan to Promote Liberty, Renew Democracy, and Restore Federalism

If you want to hear me talk about this more, I explain it in the following podcast episode:
Rewriting the Constitution: Did the Founders Screw Up the Senate?

I talked about an update I made to the Veto Powers a few weeks later.

And if you want to see my first episodes on the subject, check out the 3-parter from this past summer (2023):
Max Changes the Constitution: Part 1
Max Changes the Constitution: Part 2
Max Changes the Constitution: Part 3

Revisiting the Dirichlet-Multinomial after 10 years

I recently shared the following to my research list:
(See the latest research at localmaxradio.com/labs)

It’s been over a decade since I started building what is now the Foursquare City Guide venue ratings. This is a system applied to restaurants, cafes, parks and the like to find out what’s good. One question that came up is “how many ratings do we need before we have enough information to know the quality of a place?”

In the pursuit of an answer, I made use of the Dirichlet-multinomial probability distribution. The idea was that given a bunch of count data, I can come up with a prior probability that would apply to each row (or venue) before seeing the ratings. This tells us how much we should let the data inform our beliefs.

In Bayesian terms, this is an “informative prior”.

I found this simple model to be useful in many other situations. A good example is calculating the seasonality of different food items presented at RecSys in 2014.

Because time is expensive, I wanted a method to calculate these priors quickly. So, I found a way to efficiently compress the data, thereby simplifying the computation. I ended up implementing this in a blazing fast Python script.

To make these ideas available quickly, I shared it on arxiv and gave a talk about it at the New York Machine Learning meetup. Almost ten years later I still get contacted regarding this work, so I decided to make some updates to the code and the paper, addressing a few mistakes and making the language more precise. The remaster! 

I also created an addendum detailing the transformation of equations for Newton’s Method. There, I describe how the Hessian Matrix can be inverted efficiently in this particular situation. An interesting twist to this algorithm is doing it in "log space" - which makes sense since Dirichlet parameters are always positive.

I love sharing insights about these subjects, and I hope you enjoyed this one. Have a great summer!

Practicing my Python on Multinomial Mixture Models

I just worked to revive a fun library in my good old Bayesian package, BayesPy, related to the multinomial mixture model in https://github.com/maxsklar/bayespy

Multinomial Mixture models are much simpler than LDA but can be run in lightning speed and can give you an idea of how or whether the data is clustered. It can be run on datasets of category-count data. I think of it as K-means on a probability simplex.

Just run:

python MultinomialMixture/writeSampleModel.py -A 0.3,0.3,0.3 -m 2,2 | python MultinomialMixture/writeSampleDataset.py -N 10000 -M 500 | python3 MultinomialMixture/inferMultinomialMixture.py -K 3 -C 2

It generates a random data set and attempts to figure out the multinomial mixture on the fly. Of course, inferMultinomialMixture.py can also be run on ANY applicable dataset.

There is I'm sure really short code on pymc3 that can accomplish this as well, but it's fun to revive the pure python approach every once in a while.

9 Tips for Participating in a Public Debate

So you recently agreed to do a formal debate, and you have settled on a resolution. If you are like me, you may be thinking “WHAT have I gotten myself into?”.

I recently participated in a public debate for the “Live Free and Debate” series in Rollinsford, New Hampshire. I argued against the resolution “Monarchy is better than Representative Democracy” for a full 90 minutes.

Participating in a debate is like no other form of presentation that I’ve ever experienced. It is unpredictable, it causes you to constantly rethink your position on the fly, and inspires a slight paranoia about how the audience is following your arguments. What a mind-spinner!

Afterwards, I brainstormed ways to prepare for one of these in the future in order to have a successful and enjoyable debate performance. Future me will be using this for reference, and I hope you will find this helpful as well.

1) Your primary mission is to entertain the audience.

This first point may be controversial, but remembering this will make everyone better off. You are there to get everyone on your side and crush your opponent. Right? Not so fast!

There may come a time where you need to disseminate information in a dry and dispassionate manner as part of some formal process, but this is not one of those times. Do not forget that audience members decided to give up their valuable leisure time in their busy lives to come out and see you debate. Do they hope to learn something new? Absolutely! Are they going to learn something new if they are bored to death? Not a chance!

Think of some ways to make your topic entertaining. Is it fun? Or is it mind-bending? Are there any compelling lines that can be made about it? A good place to start with entertainment and potentially humor is to look at the aspects of the debate that make people uncomfortable. Why do people disagree so vehemently on this topic?

I can’t give an exact answer on “how” to be entertaining and what would be appropriate. That all depends on the audience, the venue, and your own personal style. But I will say that if you keep the audience entertained - or even make a reasonable attempt at doing so - much else will be forgiven.

2) Know your audience

Knowing your audience is a pretty straightforward prerequisite for any presentation. Think about the venue that you are participating in. Where are they from and what media do they consume? What are likely to be their preconceived notions about the topic? What are their concerns with supporting one side versus another?

You may come up with several different types of people that you want to acknowledge and reach. It might be helpful to think or write out these “characters” and figure out what you would like to say to each one.

3) See what your opponent (and those on the same side as your opponent) have said about the issue in the past.

Suppose that your opponent comes up with a clever new argument that you have not considered before, and you have trouble coming up with a response on the fly. This is totally understandable. You may retreat to your other arguments, or you might think of something later in the debate.

But suppose alternatively that your opponent makes the classic argument in favor of their cause, or an argument that they personally have relied upon in the past. And you have nothing to say about it? How embarrassing!

You do not have time to get an entire PhD on the topic at hand, or read stacks of fully researched books and articles, critically examining all the details in each of them. But you DO have time to check out the main points. What are the typical arguments in favor of the other side? Be sure to have an answer to those.

If your opponent has said or written publicly on a topic in the past, then this is good news for you because you can go back and read it and be sure to prepare a response. If your opponent hasn’t said anything about it - then this is also good news for you because their notions haven’t been critically examined yet (at least publicly). Either way you win!

4) Gather arguments from diverse sources

Everyone thinks a little bit differently, and everyone responds to different types of arguments. A lot of people are going to come to you before the debate and claim that they have the “killer argument” that’s going to wipe your opponent away.

Don’t fall for it! Take that argument in and use it perhaps, but consider the fact that no one person has their pulse on the entire population, or even the entirety of the specific audience. Remember your audience profiles from part 2 - talk to several people who span these profiles. Talk to some people who are not in the audience and get their views.

I am reminded of the game “family feud” and how difficult it can be to guess how 100 people will respond to even a very simple question. This is why getting a cross-sectional response from a diverse panel is going to make you more prepared for anything that comes your way.

5) Edit, Edit, Edit!

When I created consumer products as a software engineer, one of the key but often neglected components is “editing”. And when editing software, it means deleting things that you’ve written - and editing products means removing features that you’ve built!

Editing is often the difference between a successful project and an unsuccessful one. Without editing, you get feature bloat, mission creep, and technical debt. All of these fancy terms point to a failure to edit!

You will develop several different arguments in favor of your cause, and you may grow attached to some of them. But it’s important to be honest about which arguments are the strongest and the simplest. These are the ones that will most likely reach your audience, and a “throwing spaghetti against the wall until it sticks” approach is not as strong.

Some of your arguments and lines should appear on the “cutting room floor”. If you feel that you are being forced to cut something that you like for time and efficiency, this is an indication that you are doing a good job.

6) When answering audience questions, refer to ideas that you’ve already established.

Opening up the audience for questions is very dangerous. There will be people getting up on their soapboxes to give their pet theories, and there will be a lot of thinking on the fly. But there will also be a lot of insightful and tough questions.

This could be disconcerting, and I feared falling into two traps. The first is sounding like a broken record and repeating the same arguments over and over. “Why isn’t this getting through to people!?” The second is to try to make up some clever argument on the fly in order to avoid the first problem.

The reality is that most audience questions will have to do with something you’ve addressed, but you need to make a bridge between what you said and their specific objection or case. Instead of thinking “I said this earlier - what’s the problem”, think “let's take what I said earlier - and now here is how it applies to your specific question”.

This puts you in a better relationship with the audience member. Don’t forget that in order for people to properly grasp concepts, they need to see it applied to a few specific instances to get the big picture. That’s primarily what you’ll be doing during the Q&A.

7) Pick your battles.

You don’t need to respond to everything your opponent says, or everything an audience member says. If you’ve already stated an argument against it, or if it is a bit out of left field - it might be something the audience will discard as well.

With apologies for relying again on the “spaghetti against the wall” analogy - let some of the incoming spaghetti fall to the ground on its own.

8) In your conclusion, deploy the best stories to illustrate your main arguments.

The conclusion to the debate should not include new arguments - it should remind the audience of the main thrust of your arguments. You might make adjustments to this as the evening wears on, particularly in identifying the strongest examples for your arguments. But your main goal here is to summarize and to bring it all together.

It would be a mistake to make the conclusion an afterthought. In fact, it might even be helpful during your prep to think about how you envision your big ending - your final nail in the coffin - and then work backwards to your opening argument.

Don’t forget to entertain your audience and keep them engaged! Use your best stories and your best illustrative examples to tie together the arguments that you’ve already made. This will leave a big impression, and in many cases will decide the outcome of the debate.

9) Chill Out and remember good sportsmanship

Your opponent, like you, is there to take the extraordinary risk of presenting ideas and holding it up to public scrutiny. You will lay out ideas against each other, and win or lose, both sides will learn and improve (and therefore win). Keep in mind that even in a one-sided result, it is never the end of the discussion. Both sides can go back, reexamine their arguments, and continue.

You should have a lot of respect for your opponent, and never underestimate them. They put themselves in a stressful and socially-risky situation just like you did.

These are my 9 initial takeaways from the debate. What do you think? Anything that I missed?

RESEARCH: Bias Correction for Supervised Machine Learning

I am thrilled to share that after a ton of work I recently published a research paper on Arxiv called Sampling Bias Correction for Supervised Machine Learning. A local link is available as well as a Local Maximum podcast episode (218).

The paper says that if you receive only a portion of a dataset for machine learning, and you know the mechanisms by which the original data was abridged, you can work out the formulas for learning on the original dataset.

The lost data will make your results less certain, but at least the bias can be counteracted in a principled way.

I originally wrote this because of a problem that I worked out when I was building the attribution product for Foursquare in 2017. We were using machine learning to predict the likelihood of people visiting places. Because the “non-visit” examples were humongous, we sampled it at a larger rate than the “visit” examples. The mathematics of sampling bias was researched to counteract this.

I encourage anyone who is interested in machine learning and computer science in general to check out the introduction. It’s a quick read - and presents the high level ideas in our field.

Section 3 is really just “machine learning 101” which I started writing in order to establish the vocabulary, and eventually decided to make it a free-standing walk-through of the machine learning process. If you want an introduction to machine learning from a Bayesian perspective, read this section! It serves as a high-level primer for people who are unfamiliar with machine learning but have some mathematical background.

Supervised machine learning is not the only variety - and it is not always presented as a Bayesian inference problem - but it is an incredible tool for anyone trying to learn how this all works.

If you want to gain expertise in the bias correction problem, and want a deep understanding of how the Bayesian formulation solves this, then you should read the whole thing! Of course, if you need to solve the bias correction problem for work and the deadline is coming up, feel free to skip to the answers in section 5 (and 6 for logistic regression)!

I hope that this research demonstrates that formulating a machine learning problem in the language of Bayesian Inference helps to break down and answer really tough questions. This solution would not have been possible without thinking in terms of Bayesian distributions - and I think this paper will serve as an excellent case study for people to understand why that is. Once you stop thinking in terms of exact answers and start thinking in terms of beliefs over possible answers, a whole new world of insights opens up.

Finally, I want this to be used in practice.

Much noise has been made about bias in datasets. The solution I present here will allow practitioners to make assumptions about this bias and peek at the consequences of those assumptions which is a useful first step.

But more immediate is the original motivation of the paper, which is to reduce the size and composition of training sets so that computing becomes more efficient. The thinking behind this progresses as follows:

1) Big Data Product: Great news - we have a ton of data to run this model on! We can build something useful here.

2) Big Data Drawback: Hey, using all this data takes up a ton of resources. Are we past the point of diminishing returns? How about we throw out some data - pocket the savings - and the result will be just as good, or like 95% as good which is totally fine for us.

3) Uniform Random Sampling: How many examples do we have - 100 million? I think this will work on only 1 million, so for each datapoint I am going to pick a random number so that it keeps it 1% of the time. Then we'll end up with around a million points.

4) Bias Sampling: But wait!! Some data points are more valuable than others because we have a label imbalance. So let's be more selective about what we throw away. This means we can either safely throw away more data, or we can get better performance from the 1% rate we did before.

Now that bias sampling can be dealt with appropriately, this method can be deployed routinely and hopefully bring about compounding savings.

Work still needs to be done in terms of solving for specific cases (as this paper does for logistic regression) and accounting for different sampling types. Let me know if you’re interested in any of these questions - and I’ll give you my assessment!

Finally, this exercise has made me come to understand that the teaching of Bayesian Inference, while fairly standardized, is still ripe for innovation from educators and systematizers.

For example, the mathematics of Bayes rule benefits from considering probability to be proportional rather than absolute. This allows us to safely remove all of these pesky constant factors which are ultimately unnecessary and confuse everyone trying to follow the math. I tried to rely more on proportionalities and the idea of probability ratios rather than raw probabilities - but I’d like to see better notation around it. For example, in proportionality statements it should be obvious which symbols are considered variable and which are considered constant.
I hope to continue this line of research and incorporate these tools into my future projects, including newmap.ai which is currently in development.

All the Lasts

I’m optimistic that we’ll reopen, and get back to normal (with some changes in our routines and habits) at some point soon, and all of this quarantine and lockdown will be relegated to a historical curiosity that occurred at some point in our lives. But now that I have a few minutes - and maybe I’m a little bored, or a little bit wanting to procrastinate - I’ve compiled a list of “last times” for things that I won’t be able to do until this virus runs its course in the world.

Fortunately, I’m still an avid Foursquare Swarm user (as well as an employee) so I can just go to foursquare.com/history (or the single player tab in the app itself) and get all the information I want. Here it is!

Last time shopping at a “big” store: Target, March 11th. It’s been delivery and a few convenience store stops since then,

Last time at a coffee shop to sit down: Coffee Project NY, March 10th - one of the most highly rated (and also expensive) coffee shops in Brooklyn. Very enjoyable Avocado Toast. Little did I know, that was it - it’s early quarantine time! Maybe if the city shut down with us on that date, we’d be in better shape to reopen now.

Last time eating at a lunch spot: Taco Suprema, March 9th, Fort Greene Brooklyn

Last time leaving Brooklyn: March 8th, Queens

Last time at a Brooklyn Bagel Place - Brownstone Bagel and Bread Co, March 7th - nice bagels but the way I remember the place it’s pretty much impossible to do social distancing in there. This is the last day I was in a small crowd.

There was another crowd I was in that day: The Apple Store. I got the latest iPhone on March 7th as well. Feel like a different era!

Last time in Manhattan: March 6th - coming home from work after finding out they’d shut down.

Last time outside New York state: March 2nd leaving my parents in Connecticut. The toilets were out, so I had to drive to Dunkin Donuts in the area that day (which is very clean compared to the ones in Brooklny). I thought that was my major inconvenience for the year! Little did I know.

Last time eating at an Actual restaurant: Red Rose Inn, Springfield MA (March 1st on my way back from skiing).

Last time at a bar: General Stark’s Pub at Mad River Glen, Feb 29th.

Last time at a Gym that isn’t the space in front of my couch: Gramercy Place Health Club, February 24th

Last time at an in-person party: Cousin Rachel on Feb 23rd at St George’s Tavern Downtown

Last time at a sit down restaurant with other people: Parm downtown on February 18th (though I might also count Tacobi the next day, but that was a quick stop)

Last time in a movie theater: December 21, 2019. The Rise of Skywalker!

My Message To Zipcar Customer Support

Note: I’m posting this for entertainment value only, and not to get anything more than Zipcar. Despite this experience, it’s worked for me in the past - so I hope I can get it to work again. But I think this is pretty funny - and it won’t do any more damage to Zipcar than the stuff that’s already out there - so I’ll post it.

Another Note: I should post more about the better experiences in life.. but this was already written

The Message
I originally reserved the car from my apartment building garage at XYXYXYXYXYX, Brooklyn for a fun night out from 6pm to midnight.

I unexpectedly got an email saying that the reservation was moved a few blocks away to YZYZYZYZYZY lot. No Problem!

I got there at 6, and the attendants said that the car isn't there.

First call to Zipcar Customer Service: Confirmed that the previous renter is late, decided to wait. We'll give you a small credit an extra time.

Got a coffee... 6:30. Car still isn't there.

Second call to Zipcar: Ok, we'll give you a reservation in Park Slope. Oh wait that guy is late too! Ok on Henry Street.

So, we take a 15 minute uber ride to henry street. Zipcar app undoes my discount! Ok call #3 - they say the discount is coming.

Get there at 7. The car is not there! Call Uber [sic.. meant Zipcar] again.. sorry the car isn't at this location - would you like another Brooklyn location - or I can try to locate the car.

Car located! It's 5 blocks away, but you have to cross under a busy highway in an industrial zone. Run across that street!

Ok, takes a few minutes but we honk and honk and finally find the car at 7:30.

Ended up getting back at 1:45 am in some Random Part of Brooklyn! Was that really a good idea? Probably not. Wanted to get home at midnight, but got home past 2am instead because of the second Uber ride.

I posted my Uber stubbs below.. but I certify the above story is accurate. I think I deserve an extra discount - I'm not sure I should rely on Zipcar in the future after this.

Fixed Stickymap Today

Version 2 of the Google Maps API was deprecated back in 2010 - over 8 years ago! And yet for all that time, my creaking old website, stickymap.com remained functional.

I built Stickymap as an undergrad and during my first couple years out of school. In fact, I developed it further in 2008 during a multi-week leave from work similar to what I'm doing now! Stickymap was all about "sharing your neighborhood". It was developed in the 2000s - the era of web 2.0, wikis, universal broadband. Desktop and laptop computing reigned supreme. The idea was to have everyone share their locations on a map, and contribute to this great resource.

Many things have come after it - notably Foursquare - but I still love to look back on it, which is why I like to keep it up.

Well, a month or so ago it stopped worked. Google said "8 years is enough time - everyone has to get onto version 3 of the API by now". Why do they care? Well - they can charge you for version 3 for one thing. It's unclear how much I'll be charged - hopefully zero for a non-trafficked archive site.

The interesting part of today was going through a 12-year old javascript codebase - written by an amateur (me at the time) without any of the frameworks which exist today. Is this workable?

Turned out that I was able to figure out Google Maps v3 in a day. I breathed life back into the site. For now, you can explore the map and see what people posted in the past. Adding to the map and the local search feature will have to wait for another time.

I slipped in some Amazon affiliate ads - in case I get a lot of traffic and Google starts charging, I'm going to need a way to pay!

Announcing the Local Maximum Website - localmaxradio.com

I'm excited to announce today a brand new website for the Local Maximum podcast: localmaxradio.com

After producing 34 shows with a new one coming out every week, I realized that the library of content could be a wonderful resource. Each interview I give, each article I read, and each concept I discuss has been an opportunity to learn something new.

Therefore, I am launching this website as a portal to get easy access for all of this. I hope people will find this useful for education, research and reference. Here are some of the cool features of this site:

• The main page is a list of all the recent episodes: localmaxradio.com/episode

• Each episode has a show notes pages with links and an embedded SoundCloud player.

• If you want access to a single episode, and you remember the episode number, you can reach that page through a quick link. For example, if you want to access episode 1 (where we talked about Bayesian Inference applied to the Hawaii ICBM missile scare), type in localmaxradio.com/1. If you want to access episode 26 with Chris Messina, put in localmaxradio.com/26.

• These pages are easily searchable just by clicking on the magnifying glass on the upper right hand corner.

• On the topic page, localmaxradio.com/topic, you can browse all the episodes by topic. This is like an index page for the Local Maximum.

• There's also an ordered archive of all episodes, a page organized by format, and an about page.

From now on, announcements of new episodes and their show notes will appear on the new site only. The shows that are already posted on this blog won't go away, and I will continue to use this blog for other show announcements, articles I write, and any other personal messages.

Thanks to Squarespace, I was able to get this up pretty easily. I've found the process very simple - with good help videos and real-time chat help as needed. In the future, I'd like to figure out how to do some non-standard stuff like automate the SoundCloud embeddings and "Related Episodes" links on the episode pages, but that'll wait for another day.

I hope you find this helpful! If you have any feedback for the site, or any ideas for the next version - go to the contact page.

Send me Your Questions for Foursquare Founder Dennis Crowley!

Next week I get the privilege of interviewing Foursquare Founder Dennis Crowley for my new podcast, the Local Maximum*. I want to make the best of the time I have with him, so if you have a question that you'd like me to ask, please send it to me either on social media, or to localmaxradio@gmail.com.

The overall theme is technology for the next generation of consumer products, but we'll be covering other areas too. Big thought question as well as short, lightning-round type questions are welcome.

* If you read this blog, you might be saying "ugh there he goes again announcing the podcast" - but a lot of people who are clicking this link are new OK??

Cracking the last code: BYPPCVACMZQVYAP

In my last podcast episode, my quick-and-dirty algorithm for cracking a bunch of substitution cyphers made good work of all the encoded messages it was given - all except for one.

This last message goes as follows:BYPPCVACMZQVYAP

I tried tackling it by hand to no avail - with the repeated letters in the 3rd and 4th position, I could guess that the first word is "HELLO" or something like that. But I couldn't make it fit with the rest of the message!

It's very difficult to crack substitution cyphers on small messages because they contain less information, and several different substitutions of letters can make sense.

For example, consider this four-letter substitution cypher:TBBO

The only information we have about this word is that it has 3 unique letters, with the two letters in the middle repeating. It could be LOOK, BOOK, BEEP, BOOM, ROOM, or any such word like that. In this case, there's no hope in pinning down the message directly, unless we could gain more information about the word in question.

For the 15 letter message above, I think there is hope. I'm going to try a couple of ways to get ahead of it, and I'll post this to the github for my project.

1) Improve the language model.

I noticed that for short messages, it can find other messages with the same substitution pattern with lower entropy. In other words, the language model doesn't necessarily pick out the best one due to the simplicity of our language models compared to real natural language. This isn't as much of a problem with larger texts because it has so much more information to go on in separating out "real" from "fake" English.

My first thought is to move from bag-of-words to a bigram model, and work out from there.

I should also keep track of several top messages, to make sure that the real message doesn't wind up in space 2 or 3.

2) Count every possibility.

In the other cases, it wasn't possible to check every possible letter-and-space substitution because that meant checking 27! possibilities. That's a big number!

10,888,869,450,418,352,160,768,000,000

That's around 10^28. I said on the show that it has "28 zeros" as a shorthand for this order of magnitude, but of course I realized that I misspoke since it's actually not 28 zeros but 28 digits. I think most people got the point!

Anyway, in the case above, only 9 unique letters are used: ABCMPQVYZ. This means that there are far fewer substitutions to check. After selecting on of the letters as the space, there are only 8 substitutions to make! Let's do the math on that:

27! / (27-8)! = 27! / 19! = 27 * 26 * 25 * 24 * 23 * 22 * 21 * 20= 89,513,424,000

At around 89 billion possibilities that no small iteration even for a computer. And no way I'm going spin up an Amazon cluster for this thing. BUT! This can be whittled down further with some educated guesses about which letters could arise, making exhaustive search reasonable.

If you crack this one before me, either email the show localmaxradio@gmail.com, or message me on Twitter. I'll give you a shout out in the next episode!

The Local Maximum - Check out My New Podcast

Exciting news today! This is the launch day of my new podcast, “The Local Maximum”.  Yes, the day has finally arrived.

So far on my guest and solo lineup, I’ll be covering AI, Product Design, Future Technology, and Current Events. The overall blend of topics is still TBD, but I'm going to start with 10 episodes to get a handle on things.

The first episode is with my friend Aaron Bell, and we will be discussing Bayesian Inference, which will be a recurring theme on the show. Aaron also advised me in our show prep that I should have a show notes page. I’ll formalize that a bit more in the future, but for now this blog post is the show notes for episode 0.

In the software/startup world we often launch a minimal product to get the feedback loop started. This is exactly what I’m doing here. There are many things I’d like to improve. It’s a lot harder to explore complex ideas in an audio format than I thought!. But that’s my goal for this project - and I’m going to continue in the pursuit.

To send a question that could be answered on or off the show, email localmaxradio@gmail.com. I’d love to hear your opinion on the first episode, and ideas on ways to make this podcast a success.

The Local Maximum is now available on iTunes. For now, I will host the mp3s here and on soundcloud and the feed is also available on Stitcher.

The book I mentioned in the show is The Theory that Would Not Die by Sharon Bertsch McGrayne.The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy

Impress your Friends by Finding the Best Places

Designing the algorithm for Foursquare’s venue ratings is one of the best things I’ve worked on in my career. I hear people tell me that if they want to go to a good place, they make a cutoff on our 1-10 scale, say 8.5, and limit their choices to the select few elite places.

To me that sounds a little strict, but the fact remains that the Foursquare venue ratings are a great way to tell the difference between a good spot and a bad spot, and to assess the overall quality of a restaurant or bar before you go. Stephanie Yang and I spent a lot of time ensuring that our ratings are the best in the business, and I’d put these up against any venue rating system out there in terms of quality and accuracy.

Have you ever wondered how we do it? Well, we don’t give away all the secrets, but Stephanie and I wrote a blog post for the Foursquare Engineering blog called Finding the Perfect 10 where we break down some of the methods we use around venue ratings.

The Eclipse Experience

In catching up on my long backlog of potential blog posts, I want to talk about my trip this summer to see the total solar eclipse.

I knew this eclipse was coming for years, but I didn’t actually make my travel plans until a few weeks before. That wasn’t the best idea as it turns out that millions of people were visiting the same small band that stretches coast to coast to see totality. But - I made it work. I booked some flights to Louisville, Kentucky, and planned to see the eclipse down in Nashville.

Getting to Kentucky was messy; the flight was cancelled. I almost left the airport resigned to the fact that I wouldn’t be able to fly until the next day. At the last-minute, the American Airlines was able to book me a flight to Cincinnati, and from there I took an Uber to Louisville.

I had been proud of myself for coming up with the Louisville trick initially, but I had wondered whether the lack planning on my part was going to cause issues throughout the trip. Even my original shipment of eclipse glasses was cancelled and I had to rely on my second try with Amazon. Fortunately, the rest of the trip went smoothly.

For me living in New York City, it’s a treat to drive around with a rental car, particularly in a new and open area like in Kentucky and Tennessee. I enjoyed visiting the cities of Louisville, Bowling Green, and finally Nashville for the main event. As always, Foursquare directed me to the best places to see and eat. Louisville museums, Bowling Green shops and parks, and Nashville music were all great. Nashville got a little crowded with all the visitors, but if you’re willing to adjust and find backup plans it all works.

On the negative side was staying in Glasgow the night before. The hotels everywhere else had been booked far and wide. There’s really nothing there - and the bed I got wasn’t very comfortable. It was one of those cheap motels - but on that day they were charging a substantial premium. At least they had good wi-fi - I stayed up most of the night watching the latest episode of Game of Thrones - Beyond the Wall - and I was glued to my iPad. Particularly in that dragon fight beyond the wall at the end!

For many people, watching something like an eclipse isn’t worth it. It may just a natural phenomenon in the sky caused by the moon passing in front of the sun. But the event has a lot of historical and scientific significance. Ancient civilizations attached significance to this event - both positive and negative. Almost a hundred years ago in 1919, an eclipse was used to confirm Einstein’s theory of relativity. Because stars behind the sun are visible during a total eclipse, we can measure the effect of the sun’s gravity bending that light thus changing its position in the sky. Astronomers still gather data from eclipses to learn about the cosmos.

The eclipse itself - which I saw at Nashville’s Science Center - was a really cool communal experience. It started out as a really hot day, and as the more covered more and more of the sun it became cooler and cooler. The sun’s reflection on the ground - which appear through the leaves - become crescent-shaped (see picture). Finally, when all the light is blocked, you see what looks like a 360-degree sunset. It becomes dusk - and the crickets started coming out and chirping in the early afternoon! For a few minutes - everyone stopped an appreciated the view and the natural wonder that we had the privilege of catching firsthand.

I experience everything except one part. At the last-minute, a cloud passed over and I wasn’t able to see the moon-sun in totality. Fortunately, there’s another one nearby in 2024. Maybe I can plan that one a little further in advance!

Watch this Space in 2018

As the year wraps up, I just wanted to let you all know that I am planning a really exciting new project for 2018 that is going to take my "content-creation" to the next level. If you like to hear about probability, travel, technology, and all the topics I discuss - this will be something to look forward to!

I know that's cryptic - but I wanted to tease it out before the new year. In addition, I've taken 2 really fascinating trips this year that I wanted to blog about. The first was my trip to Kentucky and Tennessee to see the eclipse, and the other was my trip to the Fort Mojave Indian Reservation to do some volunteer work in that community. I was heavily involved in planning the latter which meant that this blog was a bit neglected, but I hope to give a full report on it soon!

I have a whole bunch of posts lined up too - and at Foursquare I've been shopping some talks with the "Data" crew that I may want to share publicly.

But for tonight - I'm going to go out, celebrate, and possibly have my yearly slice at Koronet Pizza. Judging from the jackets, it clearly wasn't as cold 3 years ago as it was tonight.Happy New Year everyone!

The Idea of Subjective Probability

I've been deep in Bayesian analysis recently, and I want to discuss some of the philosophical foundations.

The background here is that there are roughly two camps of statistical thought: the Frequentists and the Bayesians.  They represent very different ways of thinking about the world.  I fall squarely on the Bayesian side. The purpose of this post isn't to construct some grand argument. I just want to introduce a simple idea: Subjective Probability.

Just like the world of statistics is divided between the Frequentists and the Bayesians, the interpretation of probability is divided into objective and subjective. Objective probability is associated with the frequentists and subjective with the Bayesians.

The prime example of objective probability is a coin flip. Suppose that this is a fair coin and it produces heads on half of all flips. It is an objective property of the coin that it produces heads one out of every 2 times.

Let's look at another example: a deck of cards. A standard deck is weighted to produce a heart a quarter of the time, and to produce a picture card 3/13ths of the time. Again, it's helpful to think of the deck as yielding an objective probability - but this way of thinking is limiting.

For example, suppose you have a deck of cards on the table and you again want to assign a probability of seeing a picture card. You know that it's 3/13, but you keep staring at the top card in that deck. You see the back of that card. You know it's either a picture or it's not. "What are you?" you say. As soon as you turn over that card, the probability either goes to 0 (it's not a picture) or it goes to 1 (it is a picture).

What if you shuffled the deck and you happened to get a peak at the card on the bottom? You'd then change your expectations of what the top card is going to be. What if you caught a glimpse of that card, but you're not exactly sure?

The probability now isn't some inherent property of the deck, it's a number in your mind that represents your expectations of the top card being a picture card. This number can take into account the inherent properties of the deck of course, but it can also take into account any other information you have as well as your experience.  For example, maybe you suspect the deck is rigged. You're belief about the deck might be different from someone else's.

Subject probability applies much better in real-world forecasting situations. Let's say you want to assign a probability to a particular candidate winning an election. In the end, they'll either win or they'll lose - but the probability you assign is an expectation of that event. You don't need to be well informed to have a subjective expectation - but you want to set yourself up to have more accurate expectations as you gather more information.

Sometimes we assign binary expectations to an event. For example, if I am absolutely sure something will occur I will assign it a 1. If I believe it is impossible, I'll assign it a 0. And then I make decisions based on that belief.But it turns out that we can make better decisions by hedging. If I see on my phone that there's a 30% chance of rain, maybe I won't bring my umbrella but I'll wear clothes that I don't mind getting wet.

What does it mean to have a degree-of-belief of 30% rain? It's not like we're living in a frequentist world where that particular day can be repeated over and over again to get a fraction. This is a difficult concept to define, but another way to think about it is a ratio of expectations. If there's a 30% chance of rain, that means that there's a 70% chance of no-rain, and the ratio of expectations is 3:7. It's related to the amount of risk we're willing to take on a certain outcome.

When the event finally occurs, we can quantify how surprising that event was by using logarithms on the assigned probability of that event. For the example above, if it rains the surprise is -ln(0.3), or roughly 1.2. If it doesn't rain, it's -ln(0.7) or roughly 0.35.

Just because you're very surprised doesn't mean you were wrong to assign the probabilities that you did. It could be that your forecasting was really good given the information at hand, and a rare event occurred. But it's generally true that if you are surprised less often after adjusting your methods for assigning probabilities, your new methods are probably better. In complex systems, there's no optimal method - you can always add more data and computation. In simple games, there's usually an optimal - and these can be thought of as objective probabilities.

Anyone can assign a subject probability to an event. You'll often hear in casual conversations remarks like "there's a 20% chance we'll be on time". These probabilistic assignments are often made before any thought has been put into then. If you want to assign better probabilities, a good start is to follow some basic logic. For example, if X always leads to Y, the probability of Y must be greater than or equal to X. There's also the indifference principle: if you have no information distinguishing two mutually exclusive events, then you should assign them equal probabilities.

And finally, there's Bayes rule. This tell us how to update our beliefs when we are exposed to new information. This most important rule is how the idea of subjective probability gives rise to Bayesian inference.

I actually witnessed SQL Injection

SQL injection is one of those hacks you can do on websites with really bad security practices. It can occur whenever your database query includes user input. If the user puts something you don't expect, they can alter the database in ways that you don't expect.

A funny example - which is kind of famous in engineering circles - is given in the webcomic XKCD.

Now about 10 years ago, I coded up a site called Stickymap. It was a local search where users can post locations in their neighborhood that are interesting and leave description. It was coded in PHP. You can secure PHP if you're careful but it's very difficult to do so. If you use PHP in your organization, there should be very specific rules around running SQL queries.

biglogo

Well - one of my queries did not escape the user generated data. And, long story short someone changed every single venue name to "Bureau Veritas". Every single one. In the world.

After I investigated, I don't think that this was the intent. I think that the user was trying to add a (very spammy) description to a single venue that short-circuited the query so that the "WHERE clause" didn't make it in. For those of you who don't know, the WHERE clause in an UPDATE statement tells the database which items to update. If there is no WHERE clause, it'll update everything. Pretty insane, right?! It should probably update nothing.

I wonder how that person/spammer felt after they did this. Where they shocked? Did they move on to another site? Who knows!?

Fortunately, I had enough backup data to restore the Stickymap database while I was in San Francisco. Of course this always happens when I'm in San Francisco away from my home computer!!

Furthermore, I plugged up the security hole on the site. It's pretty cool that the security hole was left unexploited for 10 years and then all of a sudden was found. Who knows what problems we have lurking in our more critical systems? I like to hope those are more widely tested. You also want to see systems that hackers are constantly trying to exploit because that means that the owners of that system have been forced to plug the security holes. For example, I would rather trust software that's been cracked and plugged a few times in the past than software that's never been hacked but also never left out in the wild either.

Anyway - if someone out there wants to tell me there are more security holes in my site - let me know! But please try not to destroy Stickymap - it's my fun mid-2000s space on the internet and a reminder of how far we've come on local search.

And if you are the accidental culprit and you come forward, I'll either interview you for the blog, or I'll owe you a beer!

Marsbot and Chatbots

marsbot

I spoke about Marsbot a number of times on this blog, but I wanted to write my own (short) piece on what we did and why we did it. The short of it is that Marsbot is a personal assistant that tells you about all the best places around you and what to do there. The secret sauce is that you don't have to put much into it to get use out of it - you just download the app on your phone (iPhone or Android) and it automatically discovers where you go and what you like. Sometimes it'll ask you a question or two, but it also infers a lot automatically.

To get more information about it from a product perspective, I recommend that you check out both Dennis Crowley's post on Medium and also from Foursquare (and Marsbot) Product Manager Marissa Chacko.  You can also check out my talk at Talkabot in Austin. We all worked together on this for a while and are pretty psyched about the results.

Especially last December, when we got on Mashable's 12 best Apps of the year. It's nice to be on the same list as Pokemon Go - even though we far fewer users.

Now that it's been out for a while, here are a few of my takeaways from the experience.

1) Context is everything. Discoverability in the bot space isn't going to be like discoverability in the app space. There probably won't be a "bot store" and even if there is, it'll be very difficult to break through like the App stores. The winners are going to have to stand out and learn something very specific about users to help them complete a task (or have fun). Foursquare now has the Pilgrim SDK to allow other apps (and in the future hopefully bot platforms) to have the same superpowers that Marsbot has.

2) Natural Language Understanding (NLU) is the ability for a computer to understand human input. When it comes to bots, sophisticated NLU doesn't mean much unless the backend code can actually act on that understanding. For example, suppose you text Marsbot to say that the recommendation is "too far". An NLU system that gets that is only worth it if there's a backend module where Marsbot can give a closer recommendation. (There is by the way)! Therefore when it comes to bot design, I think the thing to focus on is what actions you want the bot to be able to take and expand on those. The NLU can be heuristic-based at first, and one day can be replaced by a sophisticated AI system only after a wide variety of actions are coded in the system.

3) I'm really into the conversational aspect of this. The hook for Marsbot is that it talks to you, not the other way around - but many of our users talk to Marsbot and seem to try to form a friendship with it. I imagine a seamless conversation where you can object to Marsbot's recommendations (for both places and menu items) with reasons until it comes up with a solution. I mentioned this in my talk in Austin, and some of it is implements (too far, too expensive) but Marsbot doesn't understand more than 1 command at a time. It would take a bit of work to make a fully-fleshed out human-like conversation working.

4) Marketing these bots and getting them to capture the public imagination is hard. Marsbot was lauded in the tech press, but the user numbers remain small. And even if you can build a bot with very large user numbers, how do you transition from being a fun curiosity to an indispensable tool that people rely on? I think a lot of bot-makers are doing some interesting things in the enterprise space where they can sell their technology to organizations. For the individual consumer space, the secret to the bot-hit is still elusive, but may be cracked someday!

5) You haven't heard the last of this technology from Foursquare. I think that our Pilgrim SDK will power bots like Marsbot, and our NLP + recommendation powers will continue to grow. If you're in the US, download Marsbot on your iPhone or Android device, and let me know how it goes (@maxsklar)!

Talkabot: The bot conference in Austin

Last month, I also attended the Talkabot Conference in Austin, TX. I gave a 30 minute presentation on Marsbot. I guess you could say I am on a national tour! This time, I shifted focus to how we're adopting everything we've learned about user context (location stops, taste likes, time) to send messages that are really useful.

The conference was great - it was actually my first time in Austin. I had a very warm reception from the team at Howdy and on the last day got to go out on a trek to the salt lick for some BBQ with the founders of Kip, and reps from Slack. At the conference itself, there was a lot of talk about developing standards for chat bots, and building tools and platforms upon which these bots will be built. I loved ordering coffee from a bot barista on kik who in addition to giving you coffee also pitches you his screenplays.

img_9338

Even though chat bots have existed for a while, there is a sense that we've hit an inflection point and some of the killer apps are coming. There is hope that chat could be the next great platform for innovation.

I've been asked to share my slides. Here they are: talkabot-marsbot-presentation-1

If that wasn't fun enough, Marsbot actually attended the Foursquare Halloween party last night!

Marsbot Slides for Industry Talk at RecSys 2016

I recently attended the 2016 conference on Recommender Systems at MIT with my Foursquare coworkers Stephanie Yang and Enrique Cruz. We had several contributions - 2 posters and a 20 minute industry talk on Marsbot.

Marsbot is a character in your pocket that acts as a text-based service for local recommendations. I've been working on it for a while, and we were able to do a full launch a couple months ago. I have so much to say about this project and I hope to expand on it more on this blog soon!

For now, a bunch of people have asked me to post slides from my talk at RecSys so I will post them here. I hope the video of the talk become available soon.PDF of the slides:recsys-marsbot-presentation