Predicting NFL Success By What Draft Picks Say

Thankfully, the NFL Draft and all its hype is behind us.  The matchmaking is complete but the guessing game begins as to which team picked the right combination of athletic skill, mental toughness and leadership potential in their player selections.  Hundreds of hours of game film can be broken down to grade performance with X’s and O’s.  Objective athletic tests at the NFL combine rank the NCAA football draftees by speed and strengths, just as the infamous Wonderlic intelligence test tries to rank their brain power.  

However, despite all of this data, coaches and general managers often point to a player’s set of fuzzy personal qualities, dubbed the “intangibles”, as the ultimate tie-breaking determinant to future success in the league.

Always looking for the edge in this crystal ball forecasting, teams are turning to other technologies and methods that have been used in related assessment arenas in business and politics.  As any good self-improvement speaker will tell you, success leaves clues.  By studying established leaders, certain traits, attitudes and themes can be identified as consistent “bread crumbs” left behind for others to follow.  In the same way, potential leaders that don’t pan out also demonstrate patterns of behavior that can be linked to their less-than-hyped performance.

Now, a new tool is available to NFL front offices and, as with many high-tech innovations, they have the U.S. military to thank.  Achievement Metrics, a risk prediction service for the sports industry, now provides speech content analysis meant to give the odds of a budding superstar either rising into a leadership role or sinking into legal trouble based on just their public comments.  Their base technology grew out of the work that their sister company, Social Science Automation, has provided to the CIA and government agencies including profiles of possible terrorists, based on their use of language.

Using only the transcripts from a player’s recent college press conferences or interviews, the company’s computer algorithms find patterns in a player’s words and phrases.  Its not just a few vocabulary no-no’s that set off the alarms, but rather a pattern of selected triggers from a “hot list” of over 2000 words.  So, unlike the Wonderlic IQ test that might allow for some pre-test cram sessions to increase the score, this analysis is much more intricate and based on an athlete’s words from the past.  And, by using just the transcripts of speech, the tone, volume and pronunciation of the words don’t matter; simply the ideas and subconscious selection of phrasing.

Combining numerical text analysis stats such as word meanings and frequency with established psychological profiling theories, players can be categorized in dimensions such as need for power, level of self-centerdness, ability to affect destiny and many more.

Currently, the database includes an analysis of 592 NFL players’ speech patterns matched with their off-field behavior, both positive and negative, with a correlation algorithm.  As much as this seems like a scene from Minority Report and the fictional “Pre-Crime” department, the accuracy of the results are impressive, according to the company website:

-  89 percent (89 out of 100) of the players placed in the high-risk category have been arrested or suspended while in the NFL.

-  Even more striking, only 0.13 percent (two out of 1,522) of players categorized as low-risk have been arrested or suspended during their professional careers.

-  Of the players in the database who have been arrested or suspended while in the NFL, the models placed 98 percent (104 out of 106) in the intermediate- or high-risk category based on their football-related speech from college.

Below is the current scatter plot graph that shows the distribution of NFL subjects along a “bad behavior” continuum from their database.  Any college football player who ends up in Areas 3 or 4 after his speech analysis is not good news for his future employer.

 

Here is Roger Hall, Achievment Metrics’ CEO and psychologist, explaining the process at the MIT Sloan Sports Analytics Conference held in March:

As Hall notes in his presentation, quarterbacks can have a major influence on an NFL team, so there has been much focus on the 2011 crop of draft picks and their chances of success.  Not to leave us hanging, Hall recently released the analysis of this group alongside some of the established QBs in the league.  On the Y-axis is the Positive Power score, or the level of belief in self-controlled destiny and along the X-axis is Ingroup Affiliation or the level of team orientation.  If given a choice, a team would probably prefer their prospect to be in the Aaron Rodgers/ Philip Rivers quadrant rather than the Alex Smith/Matt Leinart quadrant.

Assessing off-field risk is only the beginning for this type of analysis as long as the correlation equals causation relationship is believed and backed up with more data.  While some old school scouts and evaluators will cling to their intuitions, more forward-thinking GMs will try any new angle to get the edge.  It may just turn out to be a $20 million edge.

See also: NFL Scouting Combine Not A Good Predictor of Draft Pick Success and Advertisers Live And Die With Superstar Endorsements

NFL Scouting Combine Not A Good Predictor of Draft Pick Success

Every April, general managers and head coaches fear that their NFL Draft selection of "can't miss" college players may end up being added to the long list of past multi-million dollar draft mistakes.
So, for last month's NFL Draft, they hope they found the right matrix of information that will reveal those players with true NFL potential. One set of criteria that seems to get more media attention every year is the scouting combine, a collection of physical and mental tests given to about 300 invited prospects.

However, university researchers have now shown the tests are not good predictors of success in the NFL.

According to ESPN, of the top 10 player selections in the last five drafts (50 players total), eight have been released or traded at least once and five are completely out of the league.

Teams are becoming less willing to gamble millions of dollars on a player who has not played a single snap in the league.

The combine event, held in Indianapolis each February, was meant to provide some common denominators to compare players. Physical tests such as the 40-yard dash, shuttle and agility runs, bench press, and the vertical jump are combined with the Wonderlic Personnel Test (WPT), a 50-question general intelligence test, to paint a profile of a player beyond his on-field resume.

Of course, teams should evaluate the whole package of game film, interviews and position-specific drills, but the combine data seems to be growing in influence. A player's stock seems to rise and fall with their performance at Indianapolis.

In fact, a 2003 Arizona State University study showed that performance at the combine was directly related to draft order, which might indicate that teams rely on these tests more than they admit.

Specific combine tests also seem to make a difference in getting drafted. Last year, University of North Carolina researchers found that there were significant performance differences between drafted and non-drafted skill players in the 40-yeard dash, the shuttle runs and the vertical jump, while drafted linemen performed better in the 40-yard dash and bench press.

But in a new study, Frank Kuzmits and Arthur Adams, professors at the University of Louisville, evaluated more than 300 quarterbacks, running backs and wide receivers drafted over six seasons from 1999-2004.

They compared the players' combine performance on seven physical tests and the WPT with measures of success in the NFL. These three skill positions were chosen as they have distinct performance statistics that can be tracked (as opposed to linemen or defensive players.)

Each position used the success metrics of draft order, salaries for years 1-3 and games played for years 1-3. In addition, QB rating, yards per carry and yards per reception were measured for quarterbacks, running backs and wide receivers, respectively.

No significant link was found between combine performance and NFL success, except between 40-yard dash times and running backs. Interestingly, even the Wonderlic aptitude test did not predict NFL achievement, even though a skill position like quarterback requires a decent amount of cognitive talent. That's not to say other psychological tests would be worthless. Kuzmits and Adams cite other studies that show a player's level of self-confidence and anxiety management to be strong clues to their future accomplishments.

Of course, not all draft picks are surrounded by great teammates and some don't even get out on the field during those first few seasons. But this research showed that good or bad performance in the combine is not related to good or bad performance on the field. So, the researchers question the value of these combine tests as a draft decision support tool.

They do see a similarity between NFL teams choosing players and companies choosing employees.
"Contemporary human resource techniques could be applied to any hiring decision, including the NFL hiring process," Kuzmits told LiveScience. "Basically, teams could develop a regression equation with various success predictors weighted (college success, combine tests and interviews, awards, psychological profile, etc.). It could be done but in the end 'art' would probably trump 'science.'"

Please visit my other sports science articles on Livescience.com