Improving weak pharmaceutical research and development (R&D) productivity, the cause of much of the industry’s problems, is critical to replacing products lost to patent expiries and boosting pharmaceutical stock valuations.

While radical changes are unlikely, companies are restructuring R&D with the goal of discovering, developing and selling drugs to serve unmet medical needs. Though pharmaceutical earnings growth over the next several years is likely to be low, there is evidence that research pipelines are improving, as measured by the novelty and number of drugs in late-stage human clinical trials when compared with five to 10 years ago. Similarly, there is evidence that companies are more disciplined in their commitment to and assessment of R&D spending.

Some of today’s problems are simply a result of the high productivity of the mid-1990s, when advances in medicinal chemistry and synthesis led to the discovery of drugs serving large unmet needs in cardiovascular disease (hypertension and high cholesterol), psychosis and depression, cancer, respiratory and infectious diseases, and diabetes.

These new drugs were boosted by the advent of direct-to-consumer advertising and the managed pharmacy benefit. Rapid industry revenue growth resulting from numerous blockbusters led to poor R&D spending decisions, often based on percentage of revenue calculations that made little sense. The prevailing belief was that a higher number of compounds in development would lead to more drug approvals and fewer likely failures were terminated early. R&D financial return metrics were often non-existent and the number and cost of late-stage clinical trials ballooned in response to regulatory requirements and global operations. Finally, the climactic large pharmaceutical corporation merger and acquisition (M&A) activity of the late 1990s hurt innovation by amalgamating disparate research organizations and adding new management layers. Subsequent analysis showed a consistent rate of output by even the best companies of only one new drug per company per year and mergers did not affect this, suggesting companies had overbuilt themselves.

Scientific progress is the foundation of R&D, and it varies; the biology of cancer, for example, is far better understood today than even several years ago, while progress in neurodegenerative diseases such as Alzheimer’s has been elusive. New technologies and tools have been developed. Companies have sought greater collaboration with government, like the National Institutes of Health (NIH); universities, the source of much basic science; patient groups, who can facilitate disease awareness and patient recruitment; and among themselves, partnering their core R&D competencies and sharing risk. Companies have reduced the number of R&D sites and moved them closer to academic centers, consolidated suppliers, lowered their fixed-cost structures, relied on more third-party service providers like contract research organizations (CROs) and reduced unnecessary clinical data and clinical trial product supply overages. To appeal to payers, more clinical trials now are done against active comparators rather than placebos.

Pharmaceutical industry managements suggest that their scientists now think more about the financial impact of their decisions and are better at recruiting talent and rewarding their best performers. Companies have moved late-stage clinical trial decisions to their business units, effectively requiring them to buy or reject drugs for which only proof of concept has been established; rejects are terminated or sold. Greater discipline and accountability from management and scientists is a good thing, though there is no clear answer as to what is the most productive R&D structure and much medical and scientific advancement remains subject to chance. The promise of genomics and other technologies is great, but flat or declining global branded pharmaceutical revenues will cause continued pressure on R&D spending.

Investors should expect “evolution, not revolution” in R&D, especially given long development times. Still, gradual R&D improvement will lead to many new discoveries and products.

See more Market Insights from Columbia Management.