Can’t spell “snake oil salesmen” without 2 S

Buyers beware

Matt Patricia? Really?

“Obviously his resume speaks for itself” Nick Sirianni told the media. It does, but not in the way you want it to Nick.

As Lions head coach he was “consistently late to team meetings” and media sessions. He was notably hated by his players. They danced on his grave when he was fired.

Agents specifically named him as a problem:

The fourth-worst destination, Detroit, has the added knock of a locker room that appears dysfunctional to players looking in from the outside. “Especially with Matt ‘I think I’m Belichick but I haven’t done s—’ Patricia,” one agent said.

Adding new voices is good, but if Patricia was such a great defensive coach he wouldn’t have spent last season calling the worst looking offense in the NFL, or the season before that in a non-coaching role after crawling back to the Patriots.

Whatever Matt Patricia is selling isn’t worth buying. To paraphrase Todd McShay, you want to bring this guy into your building?

Science or scam?

A few years ago the NFL dropped the Wonderlic, which was a test of how well a person takes the Wonderlic test. Dan Marino reportedly scored a 16 (out of 50), Blaine Gabbert a 42. It was beyond useless, all the Wonderlic really did was give people negative talking points about the players who scored poorly when that information inevitably leaked out the week or so before the draft. The tell that teams didn’t actually value the Wonderlic was that even when a player did exceptionally well it was never a selling point by the team who drafted him.

A few weeks ago Daniel Jeremiah unintentionally revealed that Bryce Young scored the highest of the 2023 QBs on the “S2 Cognitive Test” which purportedly measures athletes cognitive abilities. Since Jeremiah let it slip, the S2 has become a hot topic.

in interviews with several football executives this month, S2 testing has developed a reputation so strong in the industry that it undoubtedly will affect to some degree how quarterbacks are drafted.

That reputation seems to have come from Brock Purdy, who reportedly tested extremely high, playing well for six games. Because if they valued S2 so much last year he wouldn’t have been the 9th QB and final player drafted. Now that overnight reputation may cost CJ Stroud.

The creators of S2 pushed back, claiming that the leaks were wrong and all the top QBs did well. They’ve said that the 2023 QBs is the best class they’ve ever tested. But if they all did well, then the test has little to no predictive value. In all likelihood two of the four top QBs will not work out, and it’s possible that none of them will. In which case you’ve got a test that bad QBs aced. Congratulations.

Stroud reportedly doing exceptionally bad on the S2 has brought us right back to the same place we were with the Wonderlic, where suddenly the discussion about a player isn’t about his on the field abilities but about how well he graded on a test that measures how well you take a test. The Wonderlic didn’t matter, is the S2 any different? Or are we being sold a case of snake oil?

The ability to recall information, focus on multiple things at once, and make quick decisions are relevant to playing sports; solving the water jug problem from Die Hard With a Vengeance is not. In theory, the information S2 produces has utility. But what about the reality of it? Are we sure the test is even relevant, let alone trustworthy?

In February the Athletic wrote about how Brock Purdy aced the test and repeatedly referenced how well Drew Brees did and how Purdy basically matched him. But that doesn’t tell us anything about the test. The S2 began in 2015, by then Brees had been in the league for 15 years. The article mentions Tom Brady and Aaron Rodgers but does not say if they tested, it brings them up out of nowhere and never mentions them again.

The company recently looked at 27 starting quarterbacks. (Some of the older veterans like Tom Brady and Aaron Rodgers had entered the league before S2 began testing in 2015 and there are no scores for them; Brees took the test while already playing in the NFL.) Of that group, 13 had a career passer rating above 90. The average S2 score of those players was the 91st percentile. Those with passer ratings below 90 had much lower test results.

“Those 14 guys, the average score was in the low 60s,” Ally said.

Arbitrary endpoints alert! 90 sounds impressive, but it isn’t an elite number anymore. 18 QBs had a rating of 90+ in 2022, 19 in 2021, 23 in 2020, 16 in 2019, 24 in 2018… 90 is mediocre.

Top-tier quarterbacks have the highest average scores, followed closely by safeties. That makes sense considering safeties are known as the “quarterback of the defense” and must keep an eye on multiple moving opponents.

“The average human being can keep track of about three and a half objects at a time,” Alley said. “The average safety in the NFL, it’s closer to six.”

The positions with the third-highest scores: linebacker and cornerback.

There is a causation/correlation issue at work here. Another way to summarize QBs, DBs, and LBs doing well could be “players with a lot of experience at reading and reacting test better at reading and reacting than players who do not have a lot of experience at reading and reacting.” Is it really a surprise that players who don’t line up on the line of scrimmage, where they usually have to account for multiple players, test better at this than those who do and have to primarily worry about the one man directly in front of them? Would it really be a surprise that defensive players, who are by nature reactionary, are better at one portion of a cognitive test and that offensive players, who are by nature proactive, are better at another part?

A 2019 paper by the founders finds that where you play has a correlation to how well you test:

Overall response speeds and accuracies of football players and non-athletes were comparable in each study. However, Wylie et al. (2018) found that football players, especially defensive players, excelled at interference control, and Bashore et al. (2018) found that football players, especially offensive players, excelled at impulse control.

Everything being written this month about S2 is about how well everyone tested, and according to an article from 2020 the S2 website did boast about a player who didn’t do well:

The S2 website compares two actual, unnamed NFL safeties, one of whom was drafted in the first round and one in the fourth.

The first-rounder’s S2 score: 22, including a 2 in spatial memory, a 3 in perception speed and a 40 in decision complexity. The fourth-rounder’s S2 score: 85, including a 57 in spatial memory, a 64 in perception speed and a 97 in decision complexity.

The first-rounder did not get a second contract in the NFL; the fourth-rounder got one at $14.5 million a year.

There are two problems with this. The smaller one is that neither of their descriptors are accurate. The 4th round safety is likely Eddie Jackson, who earlier that year signed a big contract, but at $11.7M a year, not $14.5M; if not then they are talking about someone who signed said contract before he could have possibly taken the S2. We’ll get to the 1st rounder in a moment.

The bigger problem is the implication that S2 correctly predicted that they would fail and succeed, and that their failure and success was because of their cognitive ability. You may think that those outcomes could have happened for any number of reasons. And you’d probably be right. The only 1st round safety on a roster from 2015-2019 who did not receive a second contract from anyone was Matt Elam, who from the beginning of the 2015 to the 2017 offseason when he was a free agent was suspended and arrested twice and missed all of the 2015 season and half the 2016 season to injury.

The S2 website—which, FWIW, has changed its content since that Daniel Jeremiah interview—currently claims, among other things, that

Tracking Capacity measures a QB’s ability to broaden his attention and see the entire field. Players with high scores have little difficulty anticipating where defenders and receivers will move on the field.

and

Distraction Control measures a QB’s ability to shield his attention and motor performance in the face of distracting stimuli (e.g., hands in the face, hearing footsteps, lineman rolling up on the back of the leg). QBs with high scores here have that steely focus and can make the throw despite the pocket collapsing around them and 100dbs of crowd noise.

That would be useful information, but the test isn’t taken on a field with defenders or crowd noise, so it does not and can not actually measure any of those abilities. It’s a 45 minute exam done on a gaming laptop. The 2019 paper conducted a test on 565 people, they used a Mac Mini, a controller, and a 17” monitor. Their own website says you can do the test on an Xbox. That’s not football.

This all sounds like something trying to be sold to us by a huckster. It’s being sold to teams that way.

Pay us before your competition does! It doesn’t help that the S2 website looks like it was designed for a company that sells products for men. Chase less. Miss less. Smash more.

Until Bob McGinn leaked the 2023 draft class on Friday, everything that had come out about S2 were controlled leaks by its creators, and nearly everything they talked about were tightly controlled PR success stories. Brock Purdy, the last player drafted, had the best score of last year’s QB class! Patrick Mahomes and Josh Allen, who weren’t the top QB in their draft, scored really well!

That’s great, but what about the guys who didn’t do well on the test, are any of them good QBs? What about the guys who did well on the test but aren’t good QBs? We don’t know, and we may never know, because if players who test poorly go on to good careers and players who test well have poor ones then the test is worthless.

We need to be highly skeptical of the S2 meaning anything. Maybe it does, but when everything is in a black box and virtually all the information that comes out has a PR spin to it, feel free to ignore it. It seems like something Matt Patricia would buy.