A consistent, systemic approach to managing and assessing a portfolio can lead to a higher yield on R&D investments.

Across industries, organizations vary widely when it comes to R&D performance. Though many factors contribute to R&D performance gaps, organizations typically ignore or overcomplicate how they manage their R&D portfolios. A set of best practices is typically a part of those organizations that manage their R&D portfolios successfully. There's no magic here. It takes commitment, time, and experience.

A systemic approach to R&D portfolio management enables you to see how the portfolio is performing—the entire portfolio, linkages across the portfolio, and drilling down to the project level.

In considering the ten best practices, keep in mind these imperatives:

  • Ask tough questions. Initiate an internal dialogue. Don't stop asking important strategic questions that hold significance for the future of your enterprise—and focus on assessment criteria that will shed light on those questions.
  • Take a systemic view. Put tools in place that will let you look at the entire R&D portfolio at any time with new eyes—including links across the whole portfolio and by discrete areas of activity.
  • Fully inform all decisions. Have all the data, information, and graphical characterizations you need to make effective decisions across the entire portfolio.
  • Know your organizational culture. Every organization has its own culture, including possible barriers to change. Manage within that culture, and have the right people, tools, and approaches to overcome impediments.
Best Practices...

1- R&D investment decisions are made as part of a systemic, structured, and carefully defined approach used across the whole organization.

John A.Parmentola is Director for Research and Laboratory Management for the U.S. Army.

By using the same approach to manage the entire R&D portfolio, the organization can see the portfolio from many perspectives. Not only can the reviewers examine how a particular project is performing against objectives and other projects, or what R&D linkages exist across the portfolio, they can also see "snapshots" of how the whole portfolio is performing in relation to important assessment criteria. This report card for the entire portfolio is extremely valuable—particularly when the process is tailored around criteria that are considered the most important investment measures for the organization.

Surprisingly, many organizations (even big R&D spenders) use only a bottom-up, project-management approach. They look at each R&D project through a "soda straw" and ignore the important strategic parameters of cross-portfolio analysis and management. This discrete approach is appropriate, but only as part of a comprehensive analysis that includes a top-down systemic view.

John T. Walker is Director, Science & Technology Management for Navigant Consulting.

From the outset, the organizational team must understand the underlying purpose for using a "systemic" approach for portfolio management. Then the process can be structured for the organization to realize that benefit. For example, one public-sector organization uses a systemic methodology as a change management tool. Because the organization was formed from a number of distinct labs, this process helps the team develop a common perspective and shared language to move the set of labs toward one ultimate goal: to be a preeminent national laboratory.

In the commercial sector, users often think of a systemic approach in terms of efficiency, effectiveness, new product development, and return on investment. Other organizations use the approach as a communications tool. And some use it for all these reasons.

2- The people who will use the R&D management system also design and implement it.
Key stakeholders from across the company help create the assessment methodology, implement the process, and participate in investment decision-making. They also build a common language or taxonomy to facilitate dialogue.

Even in a small organization, such collaboration is difficult. But in a large organization, it can be the source of tremendous frustration. Researchers, management, and funding authorities all have different perspectives—particularly in how they look at the long-term strategy for a healthy technology pipeline versus the customer's short-term needs. Funding authorities often believe they are getting "too little return on their investments." Researchers may find themselves on short leashes—unable to do the fundamental research they believe "will strengthen competitive advantages over the long haul." And the program managers are stuck in the middle. They all need to be involved—and don't forget the customers. They have a
perspective too!

[Worst Practice: Letting IT people set the rules and build the approach. IT people tend to think that R&D portfolio management can be automated—but that is wrong thinking. Research is complex and true assessment requires dialogue.]

click the image to enlarge

Looking at the effectiveness of an organization's R&D activities by sector for major industrial companies (top chart) reveals that organizations exhibit a wide range of R&D performance, with many companies showing clear opportunities for significant improvement. An organization's R&D portfolio should have the right balance of overall risk and reward (bottom chart) with careful attention made on the strategic significance of particular criteria to others.


3- The R&D approach balances purposes, timeframes, risks, and rewards.
At one end of the spectrum, where fundamental research has a long-term horizon, risk is high. Risk is high at the other end too, as an organization transitions its new products into a commercial environment. An effective portfolio methodology lets you see the trade-off between risk and reward, as well as short-, medium-, and long-term results. The ability to capture the data graphically to portray the results across the portfolio in many dimensions is critical. This allows you to assess the contributions of each discrete project, of a set of related projects across different criteria, and of the entire portfolio—by risk and reward, short- and long-term timeframes, and other dimensions.

4- Assessments are used to validate resource allocation as part of the management decision-making process.
Start. Stop. Speed Up. Slow Down. Re-scope. Organizations continually make decisions about how to allocate investment dollars across the portfolio. Decisions made in one year affect the portfolio in the next year, and so on. But without a critical assessment of the entire portfolio, organizations are hard-pressed to understand the effect a complete set of decisions has on the portfolio and its return on investment (ROI). An annual assessment captures all decision-making and its impact on the portfolio constitution and its ROI over time.

You can also see how market changes, customer demands, and company policies have affected the research emphasis. Moreover, since an organization continually risks losing "institutional memory" as key staff leave or retire, a consistent approach and a common language let stakeholders view valuable information year over year, regardless of who is at the table.

Importantly, overall assessment dimensions should touch upon some reasonable combination of value/mission/strategy/impact/benefit, feasibility/risk, and cost. Other important criteria can support the assessment—though typically, only a few are needed to answer the critical strategic questions. Additional concepts include technology maturity, competitive impact, innovation, and uniqueness of the activity.

[Worst Practice: Spending a lot of money and working hard to build a systemic approach—only to slip back into ad hoc decision-making.]

Six Key "Reasons" Why R&D Portfolio Management Can Fail
• "We've been doing that for years."
• "Portfolio analysis takes too long—and I can't invest the time."
• "I have a whole suite of tools for doing portfolio analysis."
• "I can tell you exactly how that project generates value."
• "The research we do is too diverse to think about it systemically."
• "You've got to be kidding; that program is too important to cancel."


5- Power is not in the criteria or data format, but in how you use the tool to drive high-quality discussions and decision-making.
A systemic approach to R&D planning produces a lot of data—from objective and subjective analyses of discrete projects and the portfolio as a whole. The data allow the assessors to create hundreds of "snapshots" that characterize the portfolio. But neither data nor snapshots are the complete answer.

The real answer is in the combination of the questions you are trying to answer, the data required to answer these questions, and the perspectives shared around the table by those who have a stake in the outcome. Be sure to listen to all players, regardless of function or seniority. Organizational hierarchy can get in the way of robust conversations.

[Worst Practice: An organization that underestimates how difficult portfolio analysis really is. Capturing the data is the easy part. Making sense of the data is the hard part. Some organizations will move through all the steps, but won't take the time to understand the data.]

6- Though responsive to continuous improvement, the approach does not change often or fundamentally so that year-over-year comparisons are difficult.
It's fine to make minor changes to the methodology to accommodate an evolving organization, changing external factors, or errors in the way the original methodology was constructed. But one of the values of applying a consistent process is to see how the portfolio migrates over time. If you change the methodology dramatically from year to year, you cannot compare results. The lesson is simple: Construct the methodology. Work out all bugs in the first year. Make only minor changes in subsequent years. Some organizations find it helpful to first conduct a pilot, correct the flaws, and then roll it out.

Don't Overcomplicate the Process & Analysis

Don't bring too many people to the table.
You only need 10-12 people during the actual portfolio assessment process, not 40.
Don't include too many criteria.
It is burdensome (and overly time-consuming) to include too many criteria in the portfolio assessment methodology.
Don't spend too much time over-defining the criteria and the scoring guidelines.
However, do make sure that everyone participating agrees that the important strategic questions have real meaning and significance to the future health and performance of the enterprise. Minimize wordsmithing. Initially, people will ascribe their own interpretation to key concepts like "innovation"—but along the learning curve, interpretations are put on the table and shared, and common interpretations emerge.
Don't spend months building the process.
Though you need to get the process right from the outset, don't spend too much time building it. Don't spend too little time either. Do enough to get it right the first time.

7- The systemic approach uses illustrations and graphical results to help expedite and communicate R&D priorities throughout.
Pictures are critical. People gain insights more rapidly from graphics or cross-portfolio snapshots, than from quantitative analyses or long explanations. Once the software has completed its calculations, the ability to generate a series of snapshots to illustrate the portfolio performance can help answer strategic questions and educate investment decisions. Pictures also help the research group communicate with management executives who are removed from the R&D process, but are focused on return on investment and other measures. An illustration is also a valuable communication tool in the R&D funding process.

8- Weighting factors are used to recognize the strategic significance of particular criteria relative to others.
Many criteria are used to assess a portfolio. Each over-arching dimension (benefit, risk, cost) has supportive criteria. When building an approach to R&D portfolio management, start with an understanding of your organization's goals—and determine which assessment criteria to use to measure how the organization is doing against those goals. Remember, all criteria are not created equal. So use "importance" factors to weight criteria in the assessment framework. For example, "uniqueness" may be a very important criterion for a company struggling to define itself in the marketplace.

[Worst Practice: Spending too much time worrying about the weighting factors. One organization spent so much time arguing over weighting factors that they couldn't move forward. If they become a roadblock, move on without them.]

click the image to enlarge

A graphical representation of the strengths and weaknesses of an R&D portfolio can quickly reveal where more attention should be paid to specific areas or projects. The results of these analyses should be used as a guide to decision making, not as a replacement.

9- A mechanism is in place to allow a rapid "roll-up" of information—and also to let reviewers dig deeply into the portfolio by posing key questions.
A research portfolio is complex. On average, hundreds of discrete units of activity make up the portfolio—and many horizontal and vertical linkages exist across those activities. So the portfolio methodology and analysis must enable the investigators to characterize the overall portfolio across its major dimensions, and create an efficient way to dig deeply into the portfolio, see the linkages, and assess how even one unit of activity is contributing to the portfolio success.

By reviewing a range of graphic displays, one after the other and in logical combinations, reviewers start to build an appreciation for the strengths and weaknesses of the whole portfolio. Patterns emerge. For example, certain work units in the portfolio may routinely appear in sub-optimal (underperforming) quadrants of the charts—highlighting those units for analysis. Maybe the activity is poorly designed, underfunded, or led by an inexperienced research team requiring guidance. Many times, underperformers are stars elsewhere. The ability to dig deep and roll up the analysis is critical to understanding the productive value of discrete parts and the entire portfolio—and what problem areas to address.

10- External input and participation are part of the assessment process
For fear of losing control of the decision-making process, organizations are often reluctant to involve internal or external customers in this process. That's unfortunate because organizations are biased by nature, and sometimes too close to the action to effectively gauge the quality of work under way. If you don't involve the customer or gain some kind of external perspective, bias can creep into the system and corrupt the results. Though remember that customers have a bias too: to get what they need now. So many companies use a two-step process. First, get the kinks worked out of the process and the internal organization up the learning curve to a shared perspective. Then, invite the customer to participate in the process.

[Worse Practice: Refusing to allow customers into the practice. For one organization, their thinking was so biased that they were no longer capable of evaluating their work. Over the years, they have lost tremendous market share and credibility—some of that because of their "closed" approach.]

One last note...
Some organizations overcomplicate the process and analysis. This is not rocket science. You are putting together a common language, an integrated assessment framework, and an experiential understanding of what constitutes good R&D. You can use the analysis results to create graphic illustrations that provide a strategic perspective on the entire portfolio. Over time, this approach helps strengthen the portfolio's composition and improve its overall performance-year in, year out.

John T. Walker
Navigant Consulting
John A. Parmentola
U.S. Army

About the Authors
John T. Walker is a Director of Navigant Consulting's Science and Technology Management Practice in Boston. His primary specialty lies in working with government and private sector companies to incorporate best practices in R&D portfolio analysis and management. Walker has extensive experience in the aerospace, automotive, defense, and healthcare industries, along with experience in venture capital, private equity, and corporate investment communities.

John A. Parmentola is Director for Research and Laboratory Management for the U.S. Army in Washington, D.C. He has responsibilities for the Army's Basic and Applied Research Programs of Army Research Laboratory, Army Research Institute, Corps of Engineers, and Simulation, Training, and Instrumentation Command. Parmentola was previously a science and technology advisor to the Dept. of Energy.