How To Become Nate Silver In 9 Simple Steps

Nate Silver has built one of the best models out there. It's accurate, consistent, and totally scientific. 

One advantage of being totally scientific is that his model can be replicated.

Silver, over the past five years, has explained in depth how his model actually works.

Looking through that, we now have a step-by step-process on how to be Nate Silver. 

Here's what you'll need:

  • A copy of Microsoft Excel
  • A definitive source of campaign finance data and election data
  • A vast polling database with 15 years of impeccably accurate information.

So the first of these supplies can be picked up at your local computer store and several online resources.

The last one may be difficult to find — as early as 2010 Silver was working from a database of " 4,670 distinct polls from 264 distinct pollsters covering 869 distinct electoral contests" —  but once you have it ready to go you're all set to start your FiveThirtyEight model. 

Step One: Collect all the recent polls that you can find from within a state. We'll label each of these polls by a variable like P1, P2, P3, or just generally speaking Pi

Step Two: Figure out the poll weights. Now, we need to find the weighting for each of the polls — w1, w2, w3, or more generally wi — and that's not exactly an easy task. Each wi is made up of three different values. The "recency" of the poll (we'll call this Ri), the sample size of the poll (Ni for each Pi) and the Pollster Rating for the company that did the poll (Qi for each Pi).

  • Ri is expressed as an exponential decay function. The older the poll gets, the lower this number gets. If the same polling company releases a newer poll, Ri also goes down.  We know that in 2008, the Ri was expressed at the "half-life" of the poll. 
  • Ni has to do with the sample size. The more people that the poll Pi sampled, the higher Ni gets. There are diminishing returns, though, so the rate that Ni grows slows as the sample size gets larger and larger. 
  • The Pollster rating requires the use of that massive database of polling we mentioned earlier. This analyzes the accuracy of the poll in retrospect. Better pollsters have bigger values of Qi. 

Step Three: Get Qi.  The Pollster rating, Qi, has to be determined for each pollster before we can move forward. The ways Silver does this is rather complex, but the pollster ratings are the signature element of his model so this is important. 

The way Sliver goes about developing these ratings is outlined in this blog post from 2010, but here's the gist:

The point is to see how a pollster does compared to the mean. It's much easier to call a presidential contest (avg. error 2.8 points) than a gubernatorial primary (avg. error 7.8 points). 

First, Run a regression for races where each pollster has a dummy variable. Other variables should include recency for each poll, sample size, and dummy variables abut the race. The subsequent weight that is found for each pollster dummy value is the measured skill and is called "raw score."

Next, we need to figure out a value called the Reversion parameter. If "n" is the number of previous polls from the pollster in the sample:

reversionparameter = 1 - (0.06 * sqrt(n))

Next, we regress these raw score ratings toward the mean to account for inherent luck and variance and noise. Pollsters in the National Council on Public Polls or the AAPOR Transparency initiative are considered better than non-members

Here are the values of a variable called "groupmean" that we'll need for the final calculation:

  • NCPP or the AAPOR Transparency Initiative members: -0.50.
  • Polls by telephone: +0.26
  • Polling by means of the Internet: +1.04.
Here's how we get the adjusted score:

adjscore = (rawscore * (1 - reversionparameter)) + (groupmean * reversionparameter)

Negative numbers are good pollsters, positive ones are bad. Finally, Silver calculates Pollster-induced Error. PIE= 2 + adjscore. The minimum PIE is 0, and for our purposes Qi is PIE. 

Step Four: Get the weighted polling average.  Silver doesn't say how he combines these values Ri, Ni, and Qi to form Wi, and by now that's most of what remains of his "secret sauce." We'll just multiply them. 

Now, we have a weight Wi for each poll Pi. Take the average for each state of all the weighted polls. Each pollster has a "House Effect," or a demonstrable partisan skew as compared to the running poll average. For many, this isn't a major factor, but the House effect for each poll (Hi) has to be factored in.

When n is the number of polls in the average, the Weighted Polling Average is equal to ∑[(Wi•Pi)+Hi)/ n. 

Next, do a national trend line adjustment to the weighted polling average. Nationally, if the generic Democrat has lost two points, factor that into the Weighted Polling Average. 

Finally, do a likely voter adjustment to the likely-voter adjusted weighted polling average. Nate's version is simple, just add a certain percentage to the party that is usually expected to get more votes by the expected percentage. 

This, incidentally, is the Five Thirty Eight Weighted Polling average you'll see in each Senate race and state presidential race. 

Step Five: The FiveThirtyEight Regression. Silver creates one additional "poll" that gets averaged into the mix. 

This "poll" is actually the result of a regression that brings in the major ground effects of the race — candidate experience, partisan tilt of the state, incumbency stats, et cetera.

Silver has already figured out the coefficients based on long time study of prior races, like the marginal quantitative advantage that a Governor has over a House Representative in a Senate election.

You'll have to figure that out on your own by running regressions of historical data, just like Silver presumably did. Once you do figure out those coefficients (A, B, C...), push them into a regression that looks something like this:

LR = Ax1 + Bx2 + Cx3 + Dx 4  + Ex5 + Fx6 

 LR is the result of this poll, and the xi values correspond to these variables:

  • x1, the partisan voting index of the state. This is probably taken from Cook Political Report, and is a number describing the average margin of victory for a generic Republican candidate over a generic Democrat. In Virginia, for example, a generic Republican would beat a generic Democrat by 1 point, so the PVI would be 1. 
  • x2, the Party Identification in the state. This value compares the number of registered Dems to the number of registered GOP members.
  • x3, a number (or numbers) describing fundraising data for each candidate
  • x4, a binary variable for incumbency. 
  • x5, a number describing the approval rating of an incumbent, if there is one.
  • x6, a series of dummy variables describing "stature." For senators and governors this is 3, for Representatives, Attorney Generals and big-city mayors this is 2, for State-level offices this is 1, for no prior experience this is 0. 
LR then goes into the average as a poll. It's weighted as a well-regarded poll, with a weight of 0.6. Once factored into the average, this becomes the FiveThirtyEight State Fundamentals snapshot. This is known hereinafter as the "projected vote split"
Step Six: Error analysis. The error for the snapshot projection is determined from variables based on prior FiveThirtyEight projections. Here are the ones that Silver specifically refers to as important:
  • The error is higher in races with fewer polls
  • The error is higher in races where the polls disagree with one another.
  • The error is higher when there are a larger number of undecided voters.
  • The error is higher when the margin between the two candidates is lopsided.
  • The error is higher the further one is from Election Day.

After finishing this, we have two statistics: The projected vote split and the standard error. 

Step Seven: Prepare for simulation. 

So now, we want to run around 100,000 simulations to figure out what happens next. 

We have projected vote split and standard error for each of the 50 states for our model. 

We need to split up the error into two types: National Error and Local Error. We know total error from Step Seven when we calculated the standard error.

National error is calculated from a historical analysis of poll changes between the date of the analysis and election day as well as general changes, we'll call this NE.

Here's the relationship between Local, National, and Total error:

Local Error = √ [(Total Error)2 — (National Error)2]

Step Eight: Simulate Once. Do you have Excel ready? Great. We now calculate values for Local and National error based on a normal distribution, using Microsoft Excel's NORMINV(rand(), mu , sigma) function. 

For national error, the value of mu is equal to the observed NE and sigma is equal to the standard deviation observed when calculating NE. 

In each simulation, NE is the same for each state. But, in east state we calculate local error.

  • To calculate Total Error, run NORMINV with mu equal to to the standard error developed in Step Six for each state and the related sigma.
  • National Error comes from NE.  

Then, plug these values into the Local Error formula to get local error

So now, you have a row in excel with (a) National error and (b) 50 local errors, one for each state. Take the Local Error for each state and combine it with the related projected vote split that you got in step six. This is this simulation's result, state by state

Step Nine: Repeat step eight 100,000 times, and aggregate. 

That's the first simulation. Now, do the same thing 100,000 times. In each simulation, you'll find out which party won which states.

You can then use this to figure out the electoral vote count in each simulation. Once you have 100,000 simulated electoral vote counts, average them. This is your FiveThirtyEight projection. 

For each state, you can figure out the average margin of victory. This will show you the odds for each state. You can also program a row to figure out which state was the first to push someone over 270, which will allow you to calculate the Tipping Point states. 

Now, the key thing to becoming Nate Silver is this: Make sure to do this every day for years on end and add insightful daily commentary responding to different changes in the model that you notice after coding for hours on end. Once you manage that, you can contact Mr. Sulzberger and ask when to pick up your paycheck. 

Now see five statistics problems that will change the way you see the world >



More From Business Insider

--

Get stories like this on the Yahoo app and discover more every day.
Download it now.
Loading...
  • EU likely to hand Apple Irish tax bill of over 1 billion euro - source
    EU likely to hand Apple Irish tax bill of over 1 billion euro - source Reuters - 13 minutes ago

    The European Commission will rule against Ireland's tax dealings with Apple (AAPL.O) on Tuesday, two source familiar with the decision told Reuters, one of whom said Dublin would be told to recoup over 1 billion euros in back taxes. The Commission … More »

  • Yale grad students file petition seeking union certification Associated Press - 14 minutes ago

    Yale University graduate students on Monday petitioned the National Labor Relations Board for union recognition, saying they have organized to address concerns surrounding pay and benefits and give themselves ... … More »

  • South African Treasury, Eskom row over coal contracts as government rifts deepen Reuters - 26 minutes ago

    South Africa's Treasury and state-owned Eskom clashed on Monday over a probe into coal contracts between the power utility and a company linked to the wealthy Gupta family that is accused of holding undue political sway over President Jacob Zuma. … More »

  • European privacy regulators to scrutinise WhatsApp privacy switch Reuters - 30 minutes ago

    Facebook's move to relax the privacy policy of WhatsApp, the world's most popular mobile messaging application, will be closely scrutinised, the chair of Europe's leading group of privacy regulators said on Monday.     WhatsApp, which counts more … More »

  • US alarm as Turkey warns Syrian Kurd militia of more strikes
    US alarm as Turkey warns Syrian Kurd militia of more strikes AFP News - 35 minutes ago

    Turkey warned Monday it would carry out more strikes on a Syrian Kurdish militia if it failed to retreat beyond the Euphrates River, as Washington voiced alarm over weekend clashes. Turkish forces pressed on with a two-pronged operation inside … More »

  • Berlin to require mums to come clean on 'milkmen's kids'
    Berlin to require mums to come clean on 'milkmen's kids' AFP News - 53 minutes ago

    Germany has drafted a law requiring mothers to inform their partners if their children were fathered by another man, Justice Minister Heiko Maas said on Monday. It would apply to cases in which men who believe they are not the real fathers seek … More »

  • RUBBER-Tokyo futures prices after evening session Aug 29 Reuters - 1 hour 21 minutes ago

    Following are prices for Tokyo rubber futures at the end of the Tokyo market's evening session on Aug 2016. Contract Day Volume Evening Volume * Sep 156.0 25 - - * Oct 156.0 86 156.0 1 * Nov 153.8 75 153.6 ... … More »

  • Four Nomura Investment Bankers Said to Leave as Firm Shrinks Bloomberg - 1 hour 32 minutes ago

    Jason Fertig, who led Nomura’s health-care investment-banking group in the Americas, and Matthew Upton, a banker in the firm’s global-technology team, are joining Mizuho Financial Group Inc., the people said, asking not to be identified discussing … More »

  • S&P 500 Approaches High on Consumer-Spending Data; Dollar Gains Bloomberg - 1 hour 34 minutes ago

    Japanese shares led gains among the biggest equity markets after central bank chief Haruhiko Kuroda reiterated a pledge to boost stimulus if needed. The monthly U.S. payrolls report due Friday will provide more clues on whether policy makers will … More »

  • Markets Right Now: Banks lead US stocks higher on rate hopes Associated Press - 1 hour 56 minutes ago

    The latest on developments in global financial markets (all times local): 11:45 a.m. Banks are leading stocks higher at midday as investors hope for higher interest rates. Wells Fargo, the nation's largest ... … More »

  • JPMorgan Index Move Puts $370 Billion Islamic Debt on Fund Radar Bloomberg - 2 hours 1 minute ago

    Shariah-compliant notes will be eligible for inclusion in five JPMorgan gauges from Oct. 31 and will account for between 0.35 percent to 0.99 percent of total assets on the benchmarks, according to an Aug. 18 statement by the bank. Sukuk will be … More »

  • Yuan Bears Emerge From Hibernation as Fed Threatens G-20 Calm Bloomberg - 2 hours 2 minutes ago

    Derivative markets are pointing to renewed bets on yuan depreciation, with a three-month measure of expected price swings poised for the biggest monthly increase since January. The increased pessimism comes after a period of calm that sent the … More »

  • Oil down nearly 2 percent on surging OPEC output, strong dollar
    Oil down nearly 2 percent on surging OPEC output, strong dollar Reuters - 2 hours 7 minutes ago

    Oil prices fell nearly 2 percent on Monday, snapping two consecutive days of gains, on caution over galloping Middle East crude output and a stronger dollar boosted by speculation of a U.S. rate hike by the year-end. Iraq, which has exported more … More »

  • Statoil expects 'loads of' M&A deals coming up, strategy chief says
    Statoil expects 'loads of' M&A deals coming up, strategy chief says Reuters - 2 hours 10 minutes ago

    ** Says: "If you look at the acquisition and divestment markets for upstream oil and gas on a global basis over the last five years, they have been falling to all-time lows... but there are a number of active spots, one of those is in certain parts … More »

  • Gold hits near 5-week low after Fed strikes hawkish note
    Gold hits near 5-week low after Fed strikes hawkish note Reuters - 2 hours 27 minutes ago

    Gold slid to a near five-week low on Monday after comments from top Federal Reserve officials fuelled speculation that U.S. interest rates would rise sooner rather than later, boosting the dollar. Speaking at a meeting of leading central bankers in … More »

  • Liz Weston: 5 money myths you probably believe
    Liz Weston: 5 money myths you probably believe Associated Press - 2 hours 51 minutes ago

    Managing money can be complicated, and myths are often born from people's struggles to make it simpler. But simplistic solutions can cost you instead of saving you money. If you believe any of these five ... … More »

  • BOJ Needs Massive Move to Shock & Awe, $2 Trillion Investor Says Bloomberg - 3 hours ago

    “It is how much they do, and whether they can create that kind of shock and awe at this point in the cycle,” said Mark Haefele, global chief investment officer at UBS Wealth Management, in a Bloomberg Television interview, on Monday. “They could … More »

  • Grain lower, livestock mixed Associated Press - 3 hours ago

    Grain futures were lower Monday in early trading on the Chicago Board of Trade. Wheat for Sept. delivery was 10.50 cents lower at $3.7260 a bushel; Sept. corn was 2.25 cents lower at $3.14 bushel; Sept. ... … More »

  • Singapore's GIC buys stake in Vietnam's bank AFP News - 3 hours ago

    Singapore sovereign wealth fund GIC will buy a 7.7 percent stake in Vietnam's biggest bank, both companies said in a joint statement on Monday. GIC signed a memorandum of agreement to buy 305.8 million new shares of Vietcombank, the fund's first … More »

  • Copper Drops to Lowest in Two Months on Ample Supply, China Woes Bloomberg - 3 hours ago

    Inventories tracked by exchanges in Shanghai, London and New York climbed five straight days last week to the highest since May. Barclays said in a report that a recent slump in the copper market “may be an early warning sign of weakening in China’s … More »

  • Wall St rises as hawkish Fed boosts financial stocks
    Wall St rises as hawkish Fed boosts financial stocks Reuters - 3 hours ago

    Yellen, addressing a gathering of global central bankers on Friday, said the central bank was close to meeting its goals of maximum employment and stable prices, while describing consumer spending as "solid". Yellen gave little indication of when … More »

  • Foreign experts in Egypt to probe Russian airliner crash AFP - 3 hours ago

    Russian and German experts arrived in Cairo Monday to inspect the wreckage of a Metrojet plane downed over Sinai, hoping to pinpoint how it came apart, the aviation ministry said. The Islamic State group ... … More »

  • Jualo receives additional ‘multi-million US Dollar’ in their Series A round
    Jualo receives additional ‘multi-million US Dollar’ in their Series A round e27 - 3 hours ago

    In June, the e-classifieds platform announced a Series A round led by NSI Ventures Last June, e-classifieds platform Jualo announced that it has received a “multi-million US Dollar” Series A funding led by NSI Ventures. Today, the Jakarta-based … More »

  • US stocks higher amid focus on possible Fed rate rise AFP - 3 hours ago

    Wall Street stocks were slightly higher Monday as traders focused on the US July jobs report coming later this week which could, if strong, support an early rate rise by the Federal Reserve. Fed Chair ... … More »

  • Trudeau Heads to China to Woo Trade Amid Weak Canada Growth (2) Bloomberg - 3 hours ago

    Trudeau departs Monday on a 10-day trip to Beijing, Shanghai, Hong Kong and Hangzhou, where he’ll attend the Group of 20 summit. He’ll meet with Premier Li Keqiang, Alibaba Group Holding Ltd.’s Jack Ma and Hong Kong billionaire Li Ka-Shing, … More »

 
Recent Quotes
Symbol Price Change % Chg 
Your most recently viewed tickers will automatically show up here if you type a ticker in the "Enter symbol/company" at the bottom of this module.
You need to enable your browser cookies to view your most recent quotes.
 
Sign-in to view quotes in your portfolios.

MARKET MOVERS

  • Most Actives
    Most Actives
    NamePriceChange% Chg
    0.124-0.005-3.88%
    N21.SI
    0.735-0.020-2.65%
    G13.SI
    0.128+0.014+12.28%
    RE4.SI
    0.3550.0000.00%
    E5H.SI
    4.12-0.10-2.37%
    Z74.SI
  • % Gainers
    % Gainers
    NamePriceChange% Chg
    0.355+0.115+47.92%
    BAZ.SI
    0.151+0.024+18.90%
    D79.SI
    0.23+0.03+15.00%
    AWI.SI
    0.475+0.060+14.46%
    588.SI
    0.128+0.014+12.28%
    RE4.SI
  • % Losers
    % Losers
    NamePriceChange% Chg
    0.205-0.105-33.87%
    BCZ.SI
    0.25-0.10-28.57%
    BKX.SI
    0.13-0.05-27.78%
    AYL.SI
    0.165-0.055-25.00%
    BKZ.SI
    0.745-0.130-14.86%
    564.SI

Market Data

  • Currencies
    Currencies
    NamePriceChange% Chg
    1.3586+0.0004+0.03%
    USDSGD=X
    1.5202+0.0006+0.04%
    EURSGD=X
    1.7806-0.0003-0.02%
    GBPSGD=X
    75.0020+0.32+0.43%
    SGDJPY=X
    5.7057+0.001+0.02%
    SGDHKD=X
    2.9775+0.0112+0.38%
    SGDMYR=X
    9,759.9424+11.7539+0.12%
    SGDIDR=X
    4.9130+0.0068+0.14%
    SGDCNY=X
    1.0295+0.0032+0.31%
    AUDSGD=X
  • Commodities
    Commodities
    NamePriceChange% Chg
    1,328.00+2.10+0.16%
    GCZ16.CMX
    18.83+0.18+0.96%
    SIU16.CMX
    97.68+1.89+1.97%
    ^XAU
    2.080.000.00%
    HGU16.CMX
    47.21-0.43-0.90%
    CLV16.NYM
  • Bonds
    Bonds
    TreasuryYield (%)Yield Change
    1.18-0.06
    ^FVX
    1.57-0.07
    ^TNX
    2.22-0.08
    ^TYX