Every year, they got together. Sometimes Costco’s top executives would fly to New York to meet with Kenneth Chenault, the chairman and chief executive officer of American Express, and his team. From the card company’s perch on the 50th floor of its tower in lower Manhattan, the Costco guys peered at the views of Wall Street and the Statue of Liberty. They were used to looking over a less inspiring sea of parking lots from the windows of Costco’s three-story headquarters in suburban Issaquah, Wash., decorated with Van Gogh reproductions purchased from Art.com.

When Chenault made the reverse trip to Issaquah, the Costco guys were tickled by how meticulously Amex choreographed his movements. “Ken Chenault would have an advance team come to our office before he visited,” says Paul Latham, Costco’s vice president for membership and marketing. “They planned everything—where he would enter the building, the route to the boardroom, where he’d sit at the table.” After breakfast, Chenault would often give an elaborate presentation about the performance of Amex’s Costco affinity card, using PowerPoint decks that looked like they took weeks, maybe months, to prepare. Costco just jotted down some notes for their CEO, Craig Jelinek, to talk about.

The Amex people, most of whom had MBAs, sometimes found it amusing to deal with Costco veterans who spoke about starting out stocking warehouse shelves. Less endearing was the habit Costco executives had of referring to Amex as a “vendor.” That made the Amex people seethe. After all, they represented one of America’s oldest corporations. But they smiled and said nothing, and the corporate marriage endured for 16 years.

Costco had a similar co-branded affinity card with Amex in Canada. As its contract neared expiration in 2014, Costco solicited bids from other financial institutions to see if it could get a better deal. It did, and chose Capital One and MasterCard. Chenault went all out to hold on to the discounter in the U.S. Costco’s costs to accept credit cards as part of its deal with Amex was about 0.6 percent of every purchase, people familiar with the arrangements say, which was pretty cheap for any retailer, but Chenault offered to cut them further. Still, Jelinek insisted on putting the U.S. business up for grabs, too.

Amex wasn’t happy about competing with global banks such as Citigroup and JPMorgan Chase and its archrivals Visa and MasterCard. But Chenault fought for the deal—even though his company might actually lose money in some cases when Costco customers swiped the card. As the negotiations dragged into January 2015, however, he became agitated and called his counterpart to remind him that Amex hadn’t only furnished Costco with its prestigious card; it had been Costco’s “trusted partner.” Jelinek interrupted, according to people who were briefed by Chenault about the call, and told him that as far as he was concerned, Amex was another vendor, just like the one that sold Costco ketchup. “If I can get cheaper ketchup somewhere else, I will,” he said. As rumors about the call spread, the rank and file who heard about it couldn’t believe someone from Costco had the nerve to insult Amex like that. Ketchup! Chenault called Jelinek a few weeks later to say Amex was pulling out.

Latham says Jelinek doesn’t remember making the ketchup comment, and that it was the Costco CEO who telephoned Chenault to break up. He says Citigroup and Visa offered better terms, and people familiar with the deal say Costco’s credit card costs would be virtually nothing. Costco also wanted its customers to have a Visa card that they could use in many more places than an Amex card. “The fact that Visa is the largest payments network, that acceptance for Visa cards is greater than any other cards in the U.S., that was certainly a factor,” Latham says.

On Feb. 12, Chenault broke the news to investors. The Canadian deal had prepared the market for such a possibility, but Chenault disclosed something stunning: 10 percent of the 112 million Amex cards were Costco-branded. Amex shares tumbled 6 percent that day. The magnitude of its Costco business amazed Moshe Orenbuch, an analyst at Credit Suisse who did some math and determined that Amex co-branded cards, which also included Delta, JetBlue, and Starwood Hotels, accounted for 23 percent of $1 trillion in overall card spending last year. “Amex maintains that it has the best brand,” Orenbuch says. “Yet they decided not to tell us that 23 percent of their business was actually co-branded, which means it’s using someone else’s brand. I find those two things somewhat inconsistent.”

In conference calls with investors, Chenault has said Amex ended the Costco deal for purely economic reasons. “The numbers didn’t add up,” he said in March. “We couldn’t accept their financial terms nor their contractual terms, some of which would have meant taking on more risk than we were comfortable with.” But the decision was also partially emotional, and it couldn’t have come at a worse time for Amex. Soon after the breakup with Costco, Bloomberg reports, JetBlue ditched Amex for Barclays and MasterCard. Fidelity may also be thinking about canceling its affinity card, which is furnished jointly by Amex and Bank of America, and possibly joining forces with Visa or MasterCard.

As the card deals unraveled, a federal judge ruled against Amex in a lawsuit filed by the U.S. Department of Justice arguing that merchants have the right to discourage customers from using Amex cards because its high fees unfairly drive up the price of goods. (Amex is appealing the decision.) Chenault, who’s been CEO since January 2001, when Bill Clinton was president, declined to be interviewed for this story. He’s assured investors that Amex would absorb the losses not just because it was a strong company but also because of its aura. “Our brand is globally recognized,” he said in March. He has a point: Amex is a rarefied company. Its cardholders are more affluent than users of Visa or MasterCard. In 2014, Amex members spent an average $144 per purchase, compared with $84 for Visa users and $90 for MasterCard holders, according to the Nilson Report, a publication that tracks the card industry. Because it delivers higher-spending customers, Amex charges merchants an average swipe fee of 2.5 percent per transaction. Visa and MasterCard settle for 2 percent for their credit cards.

Amex also heavily advertises to convince the public that its cardholders are superior to those who dare use other kinds of plastic, sponsoring Taylor Swift music video apps and drafting Tina Fey to portray an adorably frazzled, Liz Lemon-like shopper in ads for its new Amex EveryDay credit card.

This formula has made the company tremendously profitable. Its return on equity, a measure of a bank’s profitability, was 28 percent in the second quarter, higher than that of all its card-issuing rivals. In May, Warren Buffett, CEO of Berkshire Hathaway, the credit card company’s largest shareholder, reiterated his support for Chenault. “Amex is still a very, very special company,” he said, adding, “Ken has done a sensational job.”

But Amex is in a situation that’s becoming increasingly familiar to companies with formerly impervious brands, such as Procter & Gamble, Coca-Cola, and McDonald’s. For decades, they would charge extra for their products. Today their toothpaste, sodas, and burgers no longer have the same cachet. Worse for American Express and other credit card companies, technology is making the physical manifestation of their brand invisible. Who cares what card you have if you’re paying for your expensive meal through your smartphone’s digital wallet?

As Amex shares slumped, ValueAct Capital, a San Francisco activist hedge fund, disclosed in August that it had almost $1 billion of Amex stock. Jeffrey Ubben, the fund’s chief executive, has yet to disclose his plans. But he has a history of investing in companies and shaking things up, which has been followed by the departures of CEOs, most notably Microsoft’s Steve Ballmer. For Chenault, this may be a moment to reflect on whether he can be most helpful to Amex’s other shareholders by sticking around and fighting, or by making a graceful exit.

Chenault frequently evokes his 165-year-old company’s legacy. Three years ago, he appeared onstage at the American Museum of Natural History for what was billed as a “fireside chat” with Facebook Chief Operating Officer Sheryl Sandberg. It was a tech event, so Chenault didn’t wear his usual suit and tie. Instead, he showed up in a brown zipper-neck sweater and an open-collar shirt. Yet his dark slacks were creased, and his shoes gleamed as if they’d just been polished. “How do you innovate?” Sandberg asked him.

Smiling, Chenault replied that Amex knew a bit about the subject. “We didn’t start off in a dorm room or a garage,” he said. “Probably in a stable.” Founded in 1850, Amex transported packages from the East to the West Coast by stagecoach. It was a nice business until President William Howard Taft signed a bill in 1912 allowing the U.S. Post Office to carry packages at lower rates. Luckily, Amex had developed a product called the Traveler’s Cheque, which people could purchase in the U.S. and cash in Europe. Amex transformed itself into a global travel agency, planning footloose Americans’ trips, finding them English-speaking doctors in Europe and bottled water in Africa, and, for the right price, according to a 1956 Time cover story, organizing tours of Pygmy villages in Africa—anything to sell more Traveler’s Cheques.

In 1950, Diners Club introduced the first multipurpose charge card. Some at Amex thought their company should respond with its own card; others feared this would cannibalize the traveler’s check. After much hand-wringing, Ralph Reed, who ran Amex at the time, approved the first Amex card, which appeared in 1958. Five years later, there were a million cards “in force,” as the company likes to say.

Amex has faced trouble before. In the wake of the 1987 stock market crash, merchants started to turn away the Amex card because of its hefty swipe fees, which at the time were as high as 3.5 percent. In 1991, a group of 100 restaurateurs in Boston staged what became known as the Boston Fee Party. “There was a big recession going on,” says Steve DiFillippo, owner of Davio’s, a popular Italian restaurant in the city and one of the Fee Partiers. “The garbage guys and the meat guys were helping us out, but American Express wasn’t doing anything for us.” Visa did its best to make the situation worse with an ad campaign featuring celebrity chefs such as Wolfgang Puck and other merchants who didn’t take American Express.

Chenault led the effort to fix the problem. A native of Hempstead, N.Y., he was a graduate of Bowdoin College in Maine and Harvard Law School. In a Businessweek profile published three years before he became CEO, his college roommates and Amex colleagues described him as someone who was determined to rise to the top while maintaining tight control over his personal brand, never losing his temper or even raising his voice. He started at Amex in 1981 in the strategic planning division and applied his skills to a moribund operation that peddled jewelry and stereo equipment through the mail to cardholders. Chenault impressed his bosses by boosting sales from $150 million to $500 million and ended up in charge of travel-related services, which included the card division. He thought Amex should lower its fees for smaller merchants like DiFillippo to keep them in the fold. At the same time, he held the line with airlines that relied on Amex’s big-spending cardholders. “Ken was spectacular,” says Tom Ryder, a former Amex executive. “He was the leader of the turnaround effort. He’s not CEO for no reason.”

To the horror of elitists within Amex, Chenault pushed for blue-collar establishments such as Wal-Mart Stores to accept the card. But the most important deal was the one that the company struck in 1999 with Costco. At the time, Costco operated only 221 warehouses in the U.S. Its customers, however, who had to pay $40 a year to get in the door, tended to be affluent. “We tried to convince them that they should accept Amex,” recalls David House, a former executive at the credit card company who worked on the deal. “They kept saying, ‘It will never work. It’s too expensive.’ ” After much negotiating, Amex persuaded Costco to end its exclusive deal with Discover Financial Services and let Amex issue a Costco-branded credit card that would be the only one accepted at its warehouses. According to House, Amex didn’t have to lower its swipe fee to get the transaction done. Instead, he says, it agreed to pay Costco a bounty for every new cardholder it brought in.

At the contract signing in 1999 at Costco’s headquarters, Jim Sinegal, then the company’s CEO, seemed thrilled to be going into business with Amex. “This is a contract for life,” House recalls him saying.

The arrangement turned out to be hugely beneficial to Amex. By 2004, the number of Costco warehouses in the U.S. had risen nearly 50 percent since the alliance, to 327. Meanwhile, the total number of Amex cards rose from 46 million to 65 million. “Costco would have been a big contributor to that growth,” says Jason Arnold, a research analyst at RBC Capital Markets. Chenault, who’d become CEO three years earlier, ingratiated himself with Sinegal. Although they personified two different corporate cultures, by all accounts, the two men got along well, even when their relationship was tested by the financial crisis of 2008. Plagued by souring credit card loans, Amex won approval from the Federal Reserve to become a commercial bank, which enabled it to get $3.4 billion in funding from the Troubled Asset Relief Program. Amex also canceled cards, including some belonging to Costco members, which upset the retailer. “At that time, there was tension between the companies,” says Robert Ritchie, a former Amex manager who worked on the Costco account. “We had to mend the relationship.” The discussions were delicate; a discount chain where people bought 80 rolls of toilet paper was now able to push around the century-and-a-half-old company with the centurion logo.

Amex emerged from the crisis in a stronger position than its card-issuing rivals, which relied less on swipe fees and more on revolving credit. In Washington, there was talk that President Obama wanted Chenault in his cabinet, possibly as secretary of the Treasury, in his second term. That didn’t happen, but Chenault sounded increasingly like a politician. He became interested in marketing cards and services to people on the opposite end of the spectrum from Amex’s Platinum Card holders. In 2010, Amex paid about $300 million for Revolution Money, an Internet payments company for people without traditional bank accounts, and rebranded it Serve, which sounded more like a cause than a business.

In 2012, Amex launched BlueBird, a prepaid charge card with Wal-Mart that it promoted as a low-cost alternative to debit and checking accounts. In investor calls this year, Chenault has said those products are growing fast, but Amex has yet to release revenue figures for them, which suggests they’re underwhelming. “I would estimate that they make up 2 percent of the company’s card spending,” says David Robertson, publisher of the Nilson Report. “They aren’t moving the needle yet.”

Meanwhile, Amex’s high-end customers were seeing the value of their cards dwindle. It used to be that flashing the Amex Platinum Card with its $450 fee would get you into most airport lounges. But as airlines merged, most recently American Airlines and US Airways in 2013, Amex cardholders lost access to many of these oases because of long-standing relationships their proprietors had with rivals. Amex reacted by creating its own airport havens, called the Centurion Lounge; there were 14 as of last year. Customers like them, but Amex has had to rent expensive space in airports and provide its members with gourmet meals, massages, and whatever else it might take to keep them from cutting their cards in half. “American Express’s response has been brilliant” in the short term, says Henry Harteveldt, a travel industry analyst and founder of Atmosphere Research Group. “But does Amex want to operate a network of lounges? No.”

JPMorgan and Barclays have poached Amex’s high-spending customers with generous card offers and lower annual fees. Two years ago, Chase cards surpassed Amex in American households making at least $125,000 a year, according to Phoenix Marketing International, a wealth research firm. In July 2014, Chenault spent two days testifying in the Justice lawsuit, arguing that his competitors had stolen his company’s methods and threatened its existence. “They’ve got a billion cards,” he lamented. “I got fewer than 55 million. They got 9 million merchants. I got around 6. I mean, I am dwarfed. We are swimming in a sea of bank cards.”

Even in this moment of public self-pity, the 64-year-old CEO showed off his Amex invitation-only Centurion Card, better known as the Black Card, to white-haired U.S. District Court Judge Nicholas Garaufis, who was born in 1948. “I’ve never actually seen one of those,” Garaufis confessed.

“Your honor, if you would permit me,” Chenault said, producing his own. “This is a Black Card. It’s made out of titanium. And what it has is a set of very specialized services, so concierge type of services. So you can almost think of it as your personal aide.”

“I see,” Garaufis said. “I’m going to need a personal aide at some point.”

“Well, after the trial and everything’s done,” Chenault said.

“I don’t want to know,” Garaufis said.

“Part of my job is to persuade,” Chenault said. Never mind that even the Black Card no longer has the same mystique with the young moneyed set that Amex desperately needs to attract. In 2004, Kanye West boasted about his, memorably referring to it in a song as the “African American Express card.” But last year, Young Thug, the rap icon and influencer of the moment, rhymed in the hedonistic hit Lifestyle about having a $1.5 million spending limit on his Visa card.

In February, Garaufis ruled against Amex, opening the door for merchants to steer customers away from its cards. Amex called his decision “flawed” and promised to appeal. But credit card experts don’t expect Amex to prevail. “Swipe fees as we know them today will be cut in half over the next decade,” says Steve Mott, CEO of BetterBuyDesign, a payments consulting firm. In late September, shares of Amex had fallen 22 percent for the year, making it one of the worst performers in the Dow Jones industrial average. The company’s declining value has inspired some investors to revolt. In July, a group of shareholders sued Amex for not disclosing the size of the Costco portfolio sooner. Amex declined to comment.

The company’s woes by then had attracted the activist hedge fund ValueAct, whose purchase of a 1.1 percent stake in Amex made it the 12th-largest shareholder. Ubben, the fund’s preppy-looking founder, typically favors companies he thinks are temporarily underpriced. He hasn’t decided yet whether he’ll wage a campaign to transform Amex or simply sell the fund’s position, people familiar with his thinking have said. Ubben’s options are limited. Activists often turn a profit by pressuring companies to increase their dividend or the amount of stock they repurchase. But ValueAct can’t use those tactics on Amex because, as one of the largest U.S. banks, it’s subject to regulations that restrict how much extra cash can go to investors. The Federal Reserve must also sign off on its dividend as part of an annual stress test process. ValueAct declined to comment.

ValueAct could try to revive Amex’s stock by precipitating a management change. If the fund’s managers, who have met with Chenault since the stock purchase, decide to take that route, they might pursue a behind-the-scenes strategy similar to the one that worked at Microsoft. ValueAct announced in 2013 that it had bought almost $2 billion of the tech company’s shares and demanded a board seat. Shortly before Microsoft acquiesced, Ballmer resigned as CEO and later stepped down from the board, saying he wanted to devote more time to the Los Angeles Clippers, the basketball team he owns. Since ValueAct unveiled its stake, Microsoft’s shares have risen 52 percent. Some investors would applaud if Ubben took a similar approach at Amex. “I think you have a CEO who has overstayed his welcome,” says Credit Suisse’s Orenbuch. The challenge may be picking Chenault’s replacement. Ed Gilligan, the Amex president widely seen as his successor, died after becoming ill in May while returning from a business trip in Tokyo.

American Express has a mandatory retirement age of 72 for directors, which means the board needs to plan for succession, but Chenault doesn’t sound like he’s leaving anytime soon. In conference calls since the Costco loss, he’s said that Amex has new products and card deals in the works that will enable it to recover. He’s also planning for what happens when the Costco deal expires in March and the cards no longer work in the stores. As part of Costco’s new agreement, Amex must sell the loan portfolio to Citigroup, a deal that the two banks are still negotiating. Chenault says the company will do whatever it can to get Costco cardholders to switch to regular Amex cards. But that will be difficult. Chenault frequently boasts about his company’s powerful brand, but Costco’s may be more alluring.

That was evident late one afternoon in September at the Costco in Wheaton, Md. Beverly Patterson exited with four frozen pizzas, her favorite brand of chicken with Japanese-style fried rice, and the store’s wheat bread, which she adores, in her cart. A school librarian who lives in nearby Silver Spring, Patterson said she loves her Amex card, too, but perhaps not enough to keep it once it’s no longer good at Costco. “My choices are particular,” she said. “Costco has what I like.”

The doors slid open again, and Jane Lee, from Bethesda, emerged. Lee had just been approved for a Costco Amex card, and she was proud of it. “You have to have perfect credit,” she said. But now that Costco was going to go with Visa, she’d have to go through the whole process again. It’s irritating, but what could Lee do? “I love Costco!” she said.