Uncategorized Is Tensor to blame for the Pixel 6’s software issues and delayed updates? – Android Police
Signs point to yes
Google’s Pixel 6 is a triumph, delivering the company’s first in-house chipset. But it’s also falling behind, with multiple updates skipped or delayed, one pulled, and betas for Android 12L missing entirely. It’s not just a matter of bugs — though we’ve been keeping track. Digging through drivers and details, speaking to developers, there’s no smoking gun. But all signs point to the Pixel 6’s biggest strength also being its biggest issue: Tensor.
The Pixel 6 marks the first time Google has opted to skip using Qualcomm hardware in its Pixel phones. Here in the land of Android, Qualcomm is just about a monopoly in the flagship smartphone space (though that could start to change), and every Pixel to date has used the company’s chips, just as all its prior Nexus phones since the Galaxy Nexus have. With the rise of 5G, though, we’ve heard that Qualcomm’s prices are increasing, and that previously-easy decision is now proving more difficult to justify — though I suspect Google’s plans to roll its own chipset go back further than that.
The chip in the Pixel 6 and 6 Pro goes by the name “Tensor,” and Google designed it in collaboration with Samsung. Based on how much it seems to share with Samsung’s existing Exynos products, it might be more apt to say Google had some input, adding a couple custom processing units (for AI workloads and photos) on top of what seemed to be an Exynos reference design and modem — “semi-custom,” as Anandtech puts it. And Samsung’s involvement could be making an impact.
Custom CPU core designs are a hot topic, and at one point, they were tied with “better” products, but these days even Qualcomm’s Kryo chips are just lightly modified ARM reference designs. (Qualcomm still brands them with custom names, but when I spoke to hardware engineers at the recent Snapdragon Summit, the only model numbers that came out of mouths were ARM reference cores.) Samsung System LSI, the division of the company behind Samsung’s silicon efforts, tried to take on Apple and Qualcomm with its custom “Mongoose” cores that powered Exynos chipsets for years. But in 2019, the company downsized its CPU branch, giving up on custom CPU designs.
The company euphemistically said it was a change made to stay “competitive” in the global market, but the simple truth is by 2019, ARM reference core designs were pretty good, and developing good custom silicon is getting much harder. Samsung just wasn’t up to the challenge: Its Exynos chips were slow, hot, and inefficient compared to Qualcomm’s. It might have been a sad day for those hoping Samsung could turn things around, but nothing of value was lost. Rumors claim Samsung might try custom CPUs again in the future, but it’s used stock ARM reference designs for the last few years — soon to be augmented by the fruits of a GPU partnership with AMD.
Now, being bad at hardware doesn’t mean Samsung System LSI was also bad at software, but given its struggles, I’d wager if Samsung couldn’t design good cores for its own chipsets, it probably had some trouble supporting them at a driver level as well.
If you don’t know how the Android stack works, the simplified version is that there’s a kind of layered cake between the app-cherries on top and the hardware plate of the phone, translating everything up and down several different pipelines to show you TikTok videos of cats sneezing.
Image via Google.
The Android architecture works as pictured here. The bits a user sees, scrolls, and taps all live above that top Application Framework layer. Down at the bottom are the actual hardware drivers and HAL (hardware abstraction layer) that allow your apps and the system components talk to the hardware. These drivers (and sometimes HALs) are usually part of the so-called BSP, or board support package, that a chipset maker delivers to a phone manufacturer.
All the parts in between abstract this relationship, make it more secure, and optimize it, so the right instructions are sent to the right places. These drivers are usually provided by the hardware vendor, and though the most important bits are almost always closed-source and proprietary (meaning, not publicly documented at a source-code level), partners sometimes have access to source to help fix issues they run into.
In the case of Tensor, it’s likely that most of those drivers at the bottom are provided by Samsung. Nolen Johnson, DirectDefense inc. Hardware security consultant and developer relations manager at Lineage OS, tells us that in a “diff analysis” of the modem drivers (comparing code between two versions), The Pixel 6’s Tensor is nearly 1:1 identical with other Exynos 5123 modem drivers. Andrei Frumusanu, formerly of Anandtech, publicly confirmed that basically everything in Tensor, excluding the TPU (the “Tensor processing unit” that handles AI workloads) and ISP (Image Signal Processor, used for video and photos), is a Samsung hardware design. (It doesn’t really matter for this discussion, but Google may also be using a hardware accelerator for zram swap which Samsung leaves disabled.)
None of this outright proves that Samsung is in charge of the drivers for Tensor, but Google’s prior relationship with Qualcomm could be an indicator of its relationship with Samsung for the new chip.
As a matter of context, Johnson told me that Google had a specific arrangement with Qualcomm: Google would tell Qualcomm what it needed in a BSP, and then it would season its own secret stuff on top like a customized power HAL (which manages power states and performance) and a few other details. That simplifies development, as Qualcomm supplies most of the drivers outside a few tweaks. I should stress that Samsung Systems LSI has sold its products Qualcomm-style to other vendors in the past, like Motorola, but definitely has less experience doing it given the difference in volume and partners.
Ultimately, it isn't clear if Google has the same relationship with Samsung for the Pixel 6 that it did with Qualcomm on prior Pixels, and if Samsung is providing Google with a BSP that meets its requirements for the Pixel 6, or if Google is handling non-modem drivers independently (and if it is, it could be short-handed — more on that later). Google has been proud of its Tensor branding in public communications, downplaying Samsung’s undeniable involvement on a hardware level, and that could be an indication this relationship works differently when it comes to software.
Just like the Android stack itself, a lot of bits and pieces go into software updates, and this is a place where Google openly acknowledges the impact that third-party hardware vendors have. After all, lots of security issues happen in the interface between hardware and software — and sometimes even purely in hardware itself, circumvented later by software, as with the Meltdown Spectre vulnerabilities that dominated hardware news in 2018.
Understandably, most of you probably haven’t read the full details published by Google of a monthly patch’s security bulletins, but both Pixel Update Bulletins and Android Security Bulletins make extensive note of CVEs (a fancy way of saying security vulnerabilities) patched in a given release. In fact, the Pixel bulletins usually have a whole section (or two!) just for Qualcomm-specific fixes. That means, in addition to any functional patches that change features or fix user-facing bugs, Google makes explicit note of software vulnerabilities and hardware vulnerabilities fixed in that update.
Getting hardware manufacturers to speak on the record about their contracts and arrangements with vendors is basically impossible. Google, understandably, did not volunteer any information for this story. But engineers at other companies over the years have told me off the record that Qualcomm’s software support and driver development are exceedingly good, and now that it has its “own” chip, Google can’t rely on that anymore when it comes to the Pixel 6. Again, Samsung has sold Exynos chips to some other companies, and I’d wager it’s doing everything it can to deliver chipset drivers to customers, but Qualcomm is operating on another level. I could easily see Samsung stumbling in ways Google might not have been prepared for and taking longer to fix problems.
Now, it’s curious that Google hasn’t yet mentioned Samsung once in its Pixel Update Bulletins yet — doing so would be a tacit acknowledgment of the Samsung drivers and software assuredly at work inside the Pixel 6. But these monthly updates absolutely do contain fixes for hardware driver and closed-source binary components. Both Google and Qualcomm make a note of them, and issues with these software components could impact the timeliness of updates. If there’s a critical bug or vulnerability in hardware that needs to be fixed and, for some reason, it can’t be addressed in time for the usual first-Monday-of-the-month security patch deployment, I could see Google holding an update for affected Pixels for a short period to include it rather than waiting — and, seemingly, that’s exactly what it’s been doing.
The Pixel 6’s December 2021 update was pulled because of an issue with call connectivity. Well, the Pixel verifiably uses a Samsung Exynos modem and calls and mobile connectivity are the purview of the modem on a hardware basis. An issue there might not be a security vulnerability, and therefore wouldn’t have been mentioned in the January Pixel Security Bulletin (that’s the job of the less detailed functional patch notes, which do mention it), but it could have taken Samsung’s help to fix, and I doubt Samsung has quite as much experience being a smartphone chipset vendor as Qualcomm, so any modem-related changes could have taken longer to deliver.
On that note, let’s examine the full timeline of the Pixel 6’s update problems.
The Pixel 6 series was announced on October 19th and ended up in customer hands starting around the end of the month. Out of the box, it was running a November 2021 patch level (when we reviewed it initially, it was on October’s). Then it got a surprise bonus mid-November update to address fingerprint sensor performance — an issue the Pixel 6 series has had since launch, and which some argue it still suffers from. A mid-cycle update like that was a little unusual, but Pixels have a history of buggy launches, and we weren’t about to object to faster fixes.
Then the December update landed — or, more accurately, didn’t for most people. While it rolled out for other Pixels at the right time, it was delayed for the Pixel 6 for weeks, landing on December 13th, and even then it didn’t seem to be rolling out broadly. Some thought a wider deployment was simply delayed (on top of that original delay) or part of a staged update, while others speculated it might have been secretly pulled. Ultimately, it was pulled. Google said issues with dropped and disconnected calls were to blame (as we touched on before), and it scrubbed even the downloadable OTA and factory images off its site.
A new update was promised “by late January” — that signaled another delay for the normal first-Monday-of-the-month schedule. Then, on January 14th it landed, finally delivering both a fix for the call issue and long-awaited December Feature Drop changes for the Pixel 6 series.
I’ve been covering Pixel news since 2017 here at Android Police, and I can’t think of a single time a snafu like this has been compounded with so many delays and issues. The Pixel 6’s post-launch period has been unique for its update problems and the perceived severity of its issues.
So, what’s changed? According to Nolen Johnson, quite a lot.
From his position inside Lineage OS, Johnson knows quite a lot about both how drivers work in Android and Android’s development process. After all, the Lineage OS ROM has to shim and hack newer versions of Android into working on older, unsupported phones. The developers involved even contribute changes back, following the Android development process. And, from what he tells me, the Pixel 6 marks a departure from business as usual.
For a bit of added context, in some alternate timeline, the Pixel 6 would have been the second phone powered by Google’s Tensor. The 2020 Pixel 5 was actually supposed to be the first. Though that extra year of polish may have produced a better phone, Google is still behind schedule in other ways.
Usually, by this time of year, Android has come back to a unified code base for all chipsets and devices. However, following the release of Android 12, Johnson tells us things are still split, and Google is basically maintaining a separate track just for Tensor, even though a unification should have happened by now. Android 12L could be further complicating that, with this multiple-track issue explaining why there’s no version of Android 12L for the Pixel 6: Since Google’s already effectively maintaining three versions of Android 12 right now, that would just add a fourth. This split in code development could be responsible for the delayed updates.
“The normal development cycle was disrupted by the Whitechapel conversion, and due to that, we’re seeing these extended timelines for [updates]. I’m betting that even February will be delayed, and by March we might see a unified tag with 12L and Whitechapel.”
Google’s also actively hiring developers with experience working on low-level ARM chipset firmware. This could indicate it doesn’t have the resources currently to deal with owning the full stack in Tensor, assuming it is responsible for more of the drivers than it used to be under Qualcomm.
Part of me feels like I should have a tin-foil hat on as I write the rest of this, because there’s not a lot of conclusive proof, just a lot of circumstantial evidence. Something about the Pixel 6 seems to have caused Google’s usually predictable software development and update process to break down, and the most obvious change is Tensor.
If more devices were affected, I’d say it could be an Android 12, but the Pixel 6 is special: It’s been delayed when other phones haven’t, and it’s missing Android 12L builds when other Pixels aren’t. Google didn’t do anything sneaky like shoving a mainline Linux kernel on it (though we should point out it is running a much more recent version of the Linux Kernel than almost any other Android phone, which will help with update longevity). But, as far as we can tell, if the issues and delays are due to software, they’re tied to software unique to the Pixel 6.
I don’t know if these problems are Samsung’s fault as the almost-certain provider of driver-level software for Tensor’s near-Exynos hardware, or something on Google’s end as it learns to manage the full software stack all the way down to the silicon level for the very first time, as recent job postings indicate. Only Google can confirm that — and, so far, the company has stayed silent on the subject. But I do feel there’s enough evidence to consider Tensor a likely culprit for the Pixel 6’s delayed updates and general software woes, even if I’m confident Google can and will fix this in the long term as it stumbles through its first chipset baby.
I’m not bashing Google for switching to an “in-house” chipset (even if it’s just 90% Samsung Exynos parts and not the leap level some of us had hoped for). I think more widespread custom silicon is an inevitability for the market as the costs to differentiate and develop custom solutions inevitably come down, particularly as companies like Samsung and MediaTek start offering it as a service.
Google is wisely making an early investment, when it’s usually late to market trends. In five years, I’m confident that the 2021 Tensor will have granted Google an edge in the Android space should it continue down the custom path — hopefully with deeper changes and less mediocre drop-in Samsung bits. But right now, we’re seeing the growing pains that come from Google finally managing the full software stack down to silicon itself (or with Samsung’s less experienced help), and it’s going to take a little while for the software situation to settle into this change. And if you’re not okay with that, you don’t have to buy a Pixel 6, though we still think it’s the best Android phone.
Starting things off as a public preview
Ostensibly a senior editor, in reality just some verbose dude who digs on tech, loves Android, and hates anticompetitive practices. His only regret is that he didn’t buy a Nokia N9 in 2012. Email tips or corrections to ryne at androidpolice dot com.