Faulty software set back a bid by the world’s largest car maker for electric-vehicle dominance
ZWICKAU, Germany—Five years and nearly $50 billion into the auto industry’s biggest bet on electric vehicles, Volkswagen CEO Herbert Diess and his guest, Chancellor Angela Merkel, stood in anticipation as the first ID.3, Germany’s long-awaited answer to Tesla, rolled off the assembly line.
The event at the company’s flagship EV plant just over a year ago marked a “systemic shift from the combustion engine to the electric vehicle,” said Thomas Ulbrich, leader of the ID.3 effort.
The car, however, didn’t work as advertised.
It could drive, turn corners and stop on a dime. But the fancy technology features VW had promised were either absent or broken. The company’s programmers hadn’t yet figured out how to update the car’s software remotely. Its futuristic head-up display that was supposed to flash speed, directions and other data onto the windshield didn’t function. Early owners began reporting hundreds of other software bugs.
After years of development, Volkswagen decided in June last year to delay the launch and sell the first batch of cars without a full array of software, pending a future update, which is now scheduled for mid-February. Tens of thousands of ID.3 owners will have to bring their cars in for service to have the new software installed.
“After that the software will be regularly updated over the air,” Mr. Ulbrich said in an interview.
Volkswagen, the world’s largest car maker, has outspent all rivals in a global bid by auto incumbents to beat Tesla. For years, industry leaders and analysts pointed to the German company as evidence that, once unleashed, the old guard’s raw financial power paired with decades of engineering excellence would make short work of Elon Musk’s scrappy startup.
What they didn’t consider: Electric vehicles are more about software than hardware. And producing exquisitely engineered gas-powered cars doesn’t translate into coding savvy.
The ID.3 debacle is raising the temperature at Volkswagen. Mr. Diess nearly lost his job last year amid a revolt of Germany’s powerful IG Metall labor union and shareholder anger over the botched launch of the Golf-8, the VW brand’s breadwinner, and the bungled launch of the ID.3. He was stripped of his leadership of the VW brand, VW’s biggest business, but kept on as CEO of the entire company without day-to-day operational responsibility.
The ID.3 is gaining traction, outselling Tesla’s Model 3 in Europe in December, according to Jato Dynamics, with sales fueled by a price tag that is about $12,000 less than Tesla’s model, and by Germany’s decision last year to increase incentives for EV purchases. The ID.3 has also garnered negative trade-press reviews and is still missing key features.
Ever since Tesla launched its first car in 2008 “there was this feeling that the really serious players are going to come,” said Peter Rawlinson, CEO of electric car startup Lucid Technologies and the former chief engineer of Tesla’s Model S. Now, he says, “the Germans have finally come, and they’re not as good as Tesla.”
Other legacy car manufacturers including General Motors, Ford, Renault, Peugeot and Toyota are bringing new electric models to market this year. Failure to keep up could redraw the global auto map, costing German car makers—Volkswagen, BMW and Daimler—their leadership status in high-end products.
Mr. Diess is drawing lessons from the mistakes on the ID.3 project as he overhauls the company’s software effort to prepare for a successor model, dubbed ID.4, which goes on sale in the U.S. later this year and will be produced at first in Europe and China and next year in Chattanooga, Tenn., as well. VW says the ID.4, its first all-electric car to be sold world-wide, will deliver on its predecessor’s promises.
“In order to be successful in this new world and secure the prosperity of many people…VW must completely change,” Mr. Diess wrote in a recent LinkedIn post.
When Mr. Diess, then head of the VW brand, launched his first EV effort five years ago, he asked Fredmund Malik, an Austrian economist, to hold a “syntegration workshop” for senior brand executives. The goal, Prof. Malik said, was to persuade managers lulled into complacency by their company’s high profitability that Tesla represented an existential threat.
A second workshop was held a month later, after VW was exposed for cheating on diesel emissions. Mr. Diess wanted to use the jolt of the crisis to overcome internal opposition to electric vehicles, Prof. Malik said. It was at this meeting that VW decided to build what would become the ID.3, complete with custom software to run the vehicle and in-car apps.
Software has been running in gas-powered cars for years. An average passenger vehicle typically includes about 80 parts fitted with chips that perform discrete tasks. These chips run code that remains static over a car’s lifetime.
With the shift to electric, computing has become the heart of the vehicle, with a central processor managing the battery, running the electric motors, brakes, lights and other critical systems as well as additional features such as entertainment or heating in the seats. Just like a gas-powered car should be serviced regularly, a modern electric vehicle may receive software updates to improve safety and performance, offer new in-car services, or unlock sources of revenue for the manufacturer.
“The key here is taking this distributed system in the car, dozens if not hundreds of applications, and centralizing everything,” says Danny Shapiro, senior director of automotive at Nvidia Corp., the graphics chip maker that has become a player in self-driving car technology. “This is very complex, especially with a car where the safety level is critical. You can’t just flip a switch and be a software company.”
In the early years of the ID.3 effort, the task to code software for the car was scattered across the organization. VW’s appointment of Christian Senger, previously head of digital services and electric mobility products, as leader of VW’s entire software development, came only in 2019, months before the vehicle’s planned launch.
The group’s first task was to create a coherent organization out of the thousands of programmers spread around the group and begin to shift critical development in-house. The first major project was VW.os, an operating system for ICAS1, the car’s central computer that could be updated remotely.
Another source of complexity was that VW picked different vendors to develop different parts of its software ecosystem. To build its industrial cloud for factories, VW teamed up with Amazon Web Services. For the automotive cloud connecting its cars, it joined with Microsoft Corp. And to build ICAS1, VW turned to Continental AG , the lead partner in a team of 19 suppliers working on developing the system.
Typically, car makers order finished components from suppliers and install them in the vehicle on the assembly line. But the software for a connected car is never finished. Like an iPhone, it is constantly evolving and requires the supplier and customer to work interactively, something that VW and Continental first had to learn as the ICAS project was under way.
VW and its suppliers had to adopt new ways of working together to build the ICAS and connect the ID.3 to the cloud. They created integrated teams that met in workshops at regular intervals to assess the state of play and plot out the next steps. During these workshops, VW often placed new demands on the group as its requirements of the ID.3 evolved, Mr. Ulbrich said.
“The iPhone today is not the same product it was in the beginning. It has evolved, it is an evolutionary process. And that is the process that VW is going through now,” Mr. Ulbrich said.
The experience convinced Mr. Diess that he needed to reboot VW’s software business. In April, he brought back Prof. Malik for a three-day workshop with about 40 of his top executives. Prof. Malik said Mr. Diess posed a simple question for the group: What do we have to do to catch up with Tesla by 2024?
The CEO opened the gathering with a blistering critique of VW’s progress. He showed a slide comparing the ID.3 to the Tesla Model 3, pointing out that while VW’s car excelled in old-world features such as spaciousness and design, Tesla beat VW hands down on such metrics as battery range and advanced computing.
At the end of the workshop, the management team had the outlines of a reboot. It would produce a new fully electric and largely self-driving car by 2025, shift more resources from the company’s old business to EVs and digitization, expand battery manufacturing, and explore new revenue streams and payment systems.
As the summer 2020 launch date for the ID.3 approached, VW told Continental to focus on critical functions. By then, Mr. Ulbrich and senior VW executives concluded that the ID.3’s remote updates weren’t yet secure enough to go on the road, Mr. Ulbrich said. The updates not only changed apps and kept the navigation up-to-date, they also made changes to core functions such as the electric powertrain.
“In mid-2020 we had to make the decision that we would have to ask the first 50,000 vehicles to come to the service stations for an update,” Mr. Ulbrich said. “Updating the vehicle’s core software is a complex process and we have to make sure at any time that our vehicles are safe.”
VW didn’t make Mr. Senger available for comment.
Karsten Michels, the senior Continental engineer working on the project, said the main problem was the teams simply ran out of time. “Maybe we underestimated how much work is involved and how little we could actually rely on existing legacy software,” Mr. Michels said in an interview.
Mr. Diess restructured the software development teams. He tapped Audi CEO Markus Duesmann, poached earlier from BMW, to be VW’s new software czar.
Mr. Diess also reached outside VW for help. He discreetly asked Dirk Hilgenberg, a BMW executive and IT specialist focused on tech turnarounds, if he would replace Mr. Senger as head of the software division.
Mr. Hilgenberg said he had no hesitation. He had spent 28 years at BMW and his track record most recently included turning around production of BMW’s X-series SUV in Spartanburg, SC. In recent years, however, he said he had become frustrated by the Munich-based auto maker’s reluctance to dive headlong into electric cars.
BMW declined to comment.
“I’d been watching what Diess was doing in terms of turnaround at VW,” he said. “I really liked it because it was decisive, bold, consistent. I’ve seen nothing in the auto industry that even comes close to that.”
Mr. Hilgenberg’s first day at VW was August 1 and he immediately started work on fixing VW.os. The first version, 1.0, is a blend of open source software and custom code by Continental and VW. To fix the current glitches, VW said it would publish an update of the software, version 1.1, in February. A more advanced version—VW.os 2.0—is targeted for 2024 and will include advanced self-driving car features.
VW’s goal is to eventually build at least 60% of automotive software in-house.
The biggest challenge, said Mr. Hilgenberg, isn’t the technology, it is the mind-set of the people—their reluctance to embrace radical change until circumstances force them to.
“In the middle of success it’s not easy to understand why you need to change now,” he says.
Another component of the reboot was the Artemis project, a new in-house design team that would take the software developed by Mr. Hilgenberg’s group and integrate it in a new electric, self-driving, and internet-connected vehicle within three years.
“We fairly quickly came to the conclusion that we needed a separate unit and needed to give it the freedom to develop, a bit like a rocket,” said Alexander Hitzinger, a Porsche and Apple veteran who presented the idea to the meeting, dubbing the project “Mission T”—as in beat Tesla. The notion was so outlandish at the time that the executives eventually chose to name the project Artemis, after NASA’s planned manned mission to the moon in 2024.
“Over the past 20 years the auto industry became more integrators than developers,” said Mr. Hitzinger. “Software is written by suppliers. This was good for a while because it drives down costs but you lose control. That’s what the auto industry has to reverse now, bring in deep technical knowledge. That’s the hard part.”
To build his leadership team, Mr. Hitzinger is reaching out to tech companies and startups, bringing the automotive and technology worlds together. His catch so far includes executives from Apple, Tesla, Nio, Jaguar Land Rover, and other companies.
The small team, expected to expand to about 250 people this year, moved into their offices in December, and have begun working on preliminary designs for the new vehicle. Scrapping the conventional auto design playbook, which begins from a vehicle platform, Mr. Hitzinger says he’s putting the customer experience inside the car first.
Change is coming, Mr. Diess wrote on his LinkedIn page, and VW must move faster. “The global transformation of the industry will take roughly 10 years,” he wrote. “With or without Volkswagen.”