How Labour estimated ethnicity from surnames

In response to requests via Twitter, this post walks readers through the general method Labour used to predict the ethnicity of Auckland house buyers from their surnames. This analysis was featured in the New Zealand Herald's lead story yesterday.

Note that there are two points in this explanation where I will refuse to go into further detail, in order to protect Labour IP. The rest of this explanation has been made publicly in various venues already, so this post does not give away any new secrets.

Part 1: 2014 demographic study

Pre-election, Labour estimated the ethnicity of every person on the electoral roll, via standard Bayesian updating. There are 3.2 million people on the roll. This was one of many demographic estimates we did for everyone in the country. Most serious political parties now engage in this kind of demographic profiling.

To estimate ethnicity, we used public NZ census data on the ethnic distribution of neighbourhoods, and also used data we developed privately about the ethnic distribution of last, middle, and first names in New Zealand. We followed some advice – especially about estimating Asian ethnicities - from prominent US academic studies. I won’t be describing that process further, as that is sensitive IP for Labour.

Using these data, our base method was to estimate people’s ethnicity in a three-step Bayesian analysis:

  • Step 1: Prior: Neighbourhood ethnic distribution. New information. Lastname distribution. Posterior: Neighbourhood / lastname ethnic distribution.
  • Step 2: Prior: Neighbourhood / lastname ethnic distribution. New information. Firstname distribution. Posterior: Neighbourhood / lastname / firstname ethnic distribution.
  • Step 3: Prior: Neighbourhood / lastname / firstname ethnic distribution. New information. Middlename distribution. Posterior: Neighbourhood / lastname / firstname / middlename ethnic distribution.

This process provides a distribution of the likely ethnicities of each person in New Zealand, given their address and their full name.

The distribution covered the probability that a person was each of the following ethnicities, drawn from the level 1 and level 2 ethnic classifications from the New Zealand census: European, Maori, Pacific (not further defined), Pacific (Samoan), Pacific (Tongan), Asian (not further defined), Asian (Chinese), Asian (Japanese), Asian (Korean), Asian (South Asian), Asian (Middle East), other.

For the person-level point estimates, we used the largest single probability. That probability was typically above 0.9.

We refined these estimates further with three tweaks to account for moderate issues we encountered estimating certain ethnicities. I won’t be describing those tweaks further, because IP.

We then tested our predictions against a more-or-less-random sample of around 3,500 known New Zealanders for whom we had ethnicity data. Our best predictions, which we have used since, were 94.8% accurate.

This is an important point. Having developed our method for estimating ethnicity, we then tested it for accuracy against real world data. Only once we were satisfied it was accurate were we willing to rely on it in our work.

Part 2: Applying the predictions to housing data

To apply our general predictions, derived in part 1 above, to the Auckland housing data, we followed a two-step process.

First, we collapsed the 1.4 million Auckland-based ethnic estimates we had by surname only, as that is the only data we had in the real estate data. This allowed us to also partly leverage the earlier electoral roll-based information we gleaned from first names, middle names, and locations as part of our surname-based estimates.

Most of the surnames pointed strongly (pr>0.9) to one and only one ethnicity, although there were some examples with more mixed predictions. It created estimates such as the following (these are the real values):

Name pr(European) pr(Maori) pr(Chinese) pr(other)
JONES 0.938 0.054 0.001 0.007
HOTERE 0.048 0.887 0.000 0.065
LEE 0.481 0.027 0.400 0.092
LI 0.028 0.001 0.957 0.014

Having done that for each individual purchaser, we then summed the probabilities across all 3,922 sales in the dataset. This provided an aggregate estimate, based on the distributions of likely ethnicities in each individual sale, for the overall ethnic distribution of house buyers in Auckland.

In doing this aggregation, we tested various ways of accounting for the fact that some sales had one surname attached, while others had two or even three, accounting for multiple people with diffrerent surnames purchasing a property together. No matter how we cut those observations, the overall pattern remained within 2% of the numbers that appeared in the New Zealand Herald.

It is that overall distribution, not data cherry-picked from any particular sale, that we then compared with various other aggregate datasets about the ethnic distribution of Auckland residents, or various subsets of Auckland residents. Many of those comparisons are detailed in the Herald article and in my Public Address blog post yesterday.

Filed under: 

Commenting on this Blog entry will be automatically closed on July 16, 2015.

Party's over

When you've got a former party leader and a former party staffer taking to the media to dare each other into legal manoeuvres to gossip about recent romantic entanglements, and the other main high-profile person in the party threatening to quit, then it is done as a political force.

The worst thing for the right would now be for the Conservatives to limp on, wasting 1-2% of the vote. I'd guess National will put the pressure on now for it to all end pretty quick.

Filed under: 

Commenting on this Blog entry will be automatically closed on June 27, 2015.


DPF and Chris Bishop have conspired to come up with a new canard-of-distraction de jour. Today's trope: "Labour are the conversatives, National the progressives, and Labour is the party of No."

At the risk of a conviction for discharging a firearm in an aquaculuture container, here are some highlights,with my responses.

Labour is now the real conservative party – fearful of innovative social policy, afraid of new ideas – in short, the party which says “No” to everything.

And which party fought so hard for the Marriage Equality Bill, and brought New Zealanders their full quota of holidays, and lead the parliamentary charge against zero hours contracts? Also, which party actually has the ideas on Auckland's housing crisis? You know, the ideas National keeps trying to water down and implement badly? Labour every time.

The party has nothing to say about the social investment approach to policy, nothing to say about better public services targets and little to say about Whanau Ora.

First, I have applauded the social investment approach publicly before, although I'll admit I am concerned it loses some of its benefit when applied to healthcare issues. But the bigger point here is: how on earth is accusing Labour of saying nothing consistent with the idea it is "the party that says no to everything. Internal consistency: fail. Duh.

You won’t find many press releases from Labour on these important social reforms, or many Parliamentary questions.

Yes, shame on Labour for filling its fixed allotment of questions with the multitude of issues where the government is failing New Zealanders. Imagine the temerity of doing that! I hear the internal Labour meeting to allocate these questions is getting longer and longer, just because there are so many serious failings where the government needs to be held to account.

If Labour does have something to say, it often reverts to tired and trite clichés.

As any observer of question time know, if you replace "Labour" in that sentence with the words "John Key" it would become much more accurate.

On the new social impact bonds, Labour wailed about people “profiting” from social services. Profit already exists throughout social services.

As Eric Crampton of The New Zealand Initiative has pointed out, private hospitals profit by providing publicly funded surgery, private pharmacies profit by filling Pharmac scripts and private medical device manufacturers profit by developing better replacement hips for publicly and privately-funded operations.

Do Labour want to nationalise all pharmacies, all GPs, all midwives?

There's the DPF we know and love: old Slippery Slope Dave! If you're opposed to social bonds, *obviously* your next move is to nationalise midwives! I can hear his inner monologue now: "...then they'll own all the farms, and we know what that's called, don't we?"

Bishop's argument is pretty specious, too. He's saying, for example, that if someone makes money from baking bread that prisoners eat, that's a good enough reason to privatise the whole prison. Perhaps he should google "Serco" to see how that one is working out.

These days, National is the party of progressive, equitable, social reform. Labour is the real conservative party – saying no to everything, opposing for opposing’s sake and uninterested in new ideas.

Where to start. First, I'm sure glad to see Bishop defining "conservatism" as "saying no to everything, opposing for opposing's sake and uninterested in new ideas." I'll remind him of that in the future. Second, it's great to see National, rhetorically at least, rejecting conservatism and embracing social democracy. It's just a pity they aren't very good at it (People's Exhibit A: Nick Smith). Third, it's a mark of how deluded National is becoming that they think they can stuff Labour into "no ideas" box, given the history. Both in the last few years, and over longer history, the major pieces of progressive social reform generally come from Labour, not National.

Filed under: 

Commenting on this Blog entry will be automatically closed on June 26, 2015.

"Guns don't kill people..."

In the wake of Charleston, there is yet more awfulness from certain quarters of the American media, and yet more tragic soul-searching on the issue of gun control in America.

>a href="">Mother Jones has a good evidence-based post debunking several pro-gun myths. Here's the most important for me: it turns out that "people with more guns kill more people" which pretty much undermines the logic behind "guns don;t kill people, people kill people."

Filed under: 

Commenting on this Blog entry will be automatically closed on June 26, 2015.

Arstrong on Auckland housing

In a sobering column about the continual distractions facing National at the moment (MBIE extravagance, Saudi sand storms, Key firing blanks in Parliament), John Armstrong reserved his harshest criticism for the government's farcical Auckland housing programmes:

Meanwhile, John Key and Nick Smith combined to further tangle the already tangled web surrounding use of Crown-owned land for housing and iwi rights of first refusal if and when some goes up for sale.

Smith's bull-in-a-china-shop approach to portfolio management has its advantages in giving the perception of action. But when it comes to tackling the Auckland housing crisis, Smith seems to be running out of options. But the problem is only going to get bigger, according to the Productivity Commission's latest report. But more on that later.

Smith has admitted to reporters he is feeling the pressure. Which is why Labour is harrying him relentlessly. [...]

All this [MBIE stuff, Saudi stuff] pales into relative insignificance when placed alongside the Productivity Commission's latest missive on Auckland housing.

The commission noted that back in 2012 the Auckland Council estimated an existing shortfall of between 20,000 and 30,000 dwellings, plus need for a further 13,000 dwellings to be built each year.

Even using the council's conservative estimate of the shortfall, the commission says 46,000 new homes were needed by the end of last year.

From 2012 to 2014, however, only a few more than 14,000 dwellings were granted consents - about half of what the council estimates is required just to accommodate new demand.

In the meantime, the shortfall of dwellings in Auckland continued to grow.

The big worry was that developers were not planning to build dwellings at a rate that would erode the shortfall. The ambitious new dwelling targets in the Auckland Housing Accord, if met, would erode the backlog, but would still leave Auckland some 26,500 dwellings short by the end of 2016.

"Each year that this shortfall continues is likely to result in additional pent-up demand," the report said.

The commission's report got scant coverage, probably because elements of it were not new. Nevertheless, it should be compulsory reading for every politician, both inside and outside Auckland.

Let's see whether Smith can both (1) dig himself out his current holes, and (2) actually take the Productivity Commission's report and recommendations seriously. I seriously doubt it.

Filed under: 

Commenting on this Blog entry will be automatically closed on June 26, 2015.

The Donald and Jeb!

There's much fun to be had in the Republican Presidential primary circus, which is starting to gather some steam.

First, Donald "The Donald" Trump has again announced he's running. It took the good people of fivethirtyeight about 3 nanoseconds to uncover the data to show why he's doomed:

Huge name recognition + huge public hatred = Pack up and go home.

Second, Jeb "please don't say my last name" Bush has announced and also unveiled his campaign logo, which some have described as reminiscent of a bad musical that closes out of town, or what would happen if Ted ran for office:

Wired has all kinds of fun with it:

JEB! is fine, as far as logos go. It’s uninspired, sure, but it gets the job done. But as a piece of typography, it’s crap.

Actually, it’s worse than that.

“It’s a piece of shit, and you can quote me on that,” says design critic Steven Heller, who has written dozens of books on everything from branding and typography to infographics and posters. He is hardly alone in being so critical, though famed typographer Chester Jenkins was a bit more gentle about it. “The logo isn’t bad as a graphic, but as a piece of typography, there are some problems,” he says. [...]

First among them is the letters, which is pretty big when you consider that’s essentially all the logo is: Jeb! in Baskerville. There’s no typical (stereotypical?) American iconography—the stars and stripes, an eagle, something—to guide the eye away from the four bright red letters in a font Jenkins describes as “a not very pleasant weight” of the type cut in the 20th century. It’s just so … ordinary. The guy who designed Jeb! used an off-the-shelf font. The result is a logo that could have been created by any kid with a Mac. Now, it’s being used as a man who wants to become our president.

Filed under: 

Commenting on this Blog entry will be automatically closed on June 21, 2015.

Crosspost - Unity, success: chicken, egg?

Reposted below is a guest post I had at Public Address yesterday:

Phil Quin, one of the people linked to the new right-of-Labour group Progress, has had an article about unity published in the Herald. Here’s a flavour:

Disunity within caucus seriously undermined Labour’s credibility,” the [Labour election review] stated. This aligns neatly with a view, ubiquitous in media and political circles, that Labour is prone to destructive factional wrangling; and that MPs were so hopelessly divided under David Cunliffe that it doomed the party’s chances. I think it’s nonsense. […]

More broadly, the Unity Hypothesis posits that party unity breeds success and, without it, electoral oblivion awaits. This is backwards: it is political success that engenders unity, and the pursuit of unity at all costs is immensely damaging.

I think Phil’s misguided on both counts, mainly because he’s arguing from supposition rather than evidence.

Last year I worked on Labour’s campaign in a relatively senior role, and as a result I had access to some of our internal research, including opinion polls and focus groups.

When people in focus groups reported having voted Labour in the past, but reported planning to vote National in 2014, they were often asked: “why have you changed your mind?” In response, issues of unity, backstabbing, in fighting, and so on came up immediately, spontaneously, and repeatedly. Sitting behind the one-way glass listening to that was no fun at all.

Of course, revealing this finding is no great secret, because National had this information from its own research, the 2014 election is now well and truly over, and the research accords with the dominant view among everyone except Phil.

Secret or not, this finding is crucial. Voters were reporting that they felt disunity was a good enough reason not to consider voting Labour in 2014. They weren’t prodded into saying so; it was just their genuine perception.

In politics, remember, perception is reality.

That means there was clear evidence from the focus groups – backed by the down-survey questions in Labour’s polling – that “disunity” really was a cause of vote loss for Labour in 2014. Whether that disunity was perceived or real is decidedly secondary. In fact, I agree with Phil there wasn’t much actual disunity in 2014. What matters is it caused vote-loss anyway.

This evidence seriously undercuts Phil’s claim that disunity had nothing to do with Labour’s 2014 performance. The voters say it did.

But what of his second point, that seeking unity as a path to success is bass-ackwards? Phil thinks instead that success is the path to unity.

Well, the voters get a say there, too. And the voters told our researchers over and over again that, from their perspective, a perceived lack of unity was a barrier between Labour and success.

Success and unity are in a chicken-egg relationship. They’re symbiotic. They’re endogenous. Success will breed further unity; I think Phil is right about that. But any disunity now makes that initial dose of success ever harder to achieve. I don’t need suppositions or arguments from personal experience to make that argument – I just asked the voters.

That brings me to a broader point about Progress. Last week I welcomed Progress on to the scene, on the basis that another voice promoting positions left of the current government is a good thing. I took comfort in Stuart Nash’s view that Progress wasn’t a new faction within Labour.

I sure hope Phil – who likes the idea of factions within Labour – read Stuart’s comments carefully. His column suggests perhaps he should give Stuart a call.

If Progress ends up being merely a group that engages in internal Labour debates via newspaper columns, the only winners will be National. I remain hopeful it won’t go that way.

Filed under: 

Commenting on this Blog entry will be automatically closed on June 20, 2015.

Oh dear, Nick

Nick Smith is in freefall. Fresh from chasing me down the comments section of the NBR to defend his honour, now he's blaming TV3's Brook Sabin for ruining his great housing policy by, you know, asking questions. He seriously did that. And he's blaming his officials. And he's blaming Ngati Whatua and Tainui for, youou know, insisting on their legal rights. And he's blaming Labour.

But the only person Nick Smith should blame looks at him every morning in the mirror.

It's Pythonesque.

Getting new housing supply in Auckland is a big task. Nick Smith is not up to that task. He has to go. Every day he stays housing Minister, the Kiwi dream of home ownership slips away for more and more Auckland families. They deserve better.

John Key: Do Auckland families a favour and bring in a Minister who's up to the task.

Filed under: 

Commenting on this Blog entry will be automatically closed on June 20, 2015.

Nick Smith, busy guy

As keen readers know, on Friday I wrote an open letter to Nick Smith calling on him to resign for completely ruining a pretty good idea by being incompetent. The NBR republished it (paywalled, so no link unfortunately).

In the comments to my letter, Nick Smith replied!

I have two points here: First, if the charge against is that you – a very busy Minister of the Crown – are doing a poor job in your portfolio, then charging around media comments sections breathlessly defending your personal honour is a pretty awful look. It actually feeds the impression Smith is seeking to suppress, that Nick Smith isn’t focused on the task at hand.

Second, let’s go bit by bit through Smith’s response:

Robs claims are false. Firstly, the Tamaki Collective representing Auckland's 13 iwi were consulted a month before the Budget announcement. This does not mean all iwi are happy but we are committed to an ongoing dialogue.

Well, that makes it worse! Now Smith’s story is he consulted with Iwi, who told hism they were pissed off and could take it all to court. Then, armed with that knowledge, he went straight over their heads to announce the sales anyway, dismissing their concerns in the House with by selectively quoting a law, quite deliberately leaving out the clause they’ll hang him on later.

Doing all that opened him up to even more court action, with judicial reviews of his post-consultation decisions. Yet again, Smith’s actions made an already bad situation worse.

Secondly four sites were initially announced - one as definitive and 3 as potential requiring more work. Subsequent revelation that one of 3 potential sites at Wiri is owned by NZTA and Auckland Council changes little - it is public land to be offered for housing development through process. Rob overlooks that AC were part of launch and both crown and council said a number of sites were jointly owned and we would do together.

Good to see even further backing down here. We started with 430 hectares of land, then we got to 30 hectares of land ready to go, and now we’re at only around a quarter of that 30 hectares is actually ready to go, the others are just ideas at this point. I’ll leave it to the journalists who were on the Nick Smith fun bus to decide whether his sales pitch at the time was actually as cautious as it appears now…

The problem here is expectations. The announcement was only made in budget late May and $2million operational and $53milllion capital only available to support detailed work from 1st July. We have said we will have first contract by October and first homes late 2016 and we will deliver that. Nick Smith

If the problem is expectations, Minister, you’d really have to point the finger at the person who built up those expectations. Who was that person? I’ll give you a clue – YOU!

Who went out there with a claim of 500 hectares of land ready to go? You did.

Who said there were no issues about the land being currently is use for incompatible purposes? You did.

Who claimed the issues with the Iwi RFR process amounted to nothing to worry about? You did.

Time and again, Nick Smith, you’ve made your situation worse by opening your big mouth. Please just go away, so Auckland families looking for a home can actually have a Minister who can actually help them. They’ll be impressed with the change, I’m sure.

Filed under: 

Commenting on this Blog entry will be automatically closed on June 19, 2015.

Today's WhaleOil lie about me, redux.

As readers know, I like to correct lies told about me by other bloggers. Here's the embattled Whaleoil:

Rob Salmond, who declared to us all during the election that Labour’s own polling was 10 percent better than the public polls – clearly intimating that the public polls were wrong and Labour was right – thinks the Progress group are on the right track.

I never, ever said Labour's internals were 10 points better than the public polls. That's a lie. What I did do was talk back at Cam's BFF Judith Collins on Twitter when she invented some Labour internal polling numbers. I guess lying about me over and over is Cam's version of giving back double for that. Oh noes!

At a lower level, I think I'd characterise my position on Progress as "a welcome addition to the debate" rather than "are on the right track," but that's semantics rather an untruth

Cam goes on:

I don’t agree with him [Rob, me] on Labour being a broad church. It certainly used to be, but under Helen Clark it became highly factionalised.

Oh, sure, Labour wasn't a broad church at all with Helen in charge, right? Oh, except for those three elections they won with an average of 40% support. Apart from that, they Helen Clark's Labour had very narrow appeal.

Filed under: 

Commenting on this Blog entry will be automatically closed on June 19, 2015.