NIAMS Training Grant and Career Development Award Program Evaluation

NIAMS conducted an outcome evaluation to assess the success of postdoctoral research trainees who received NIAMS grants and awards through its extramural research training and career development award program. Like other NIH training and career development grants and awards programs, the NIAMS program is intended to help ensure that a diverse and highly trained workforce is available to assume leadership roles related to biomedical and behavioral research. NIAMS' overall objective is to use a combination of institutional training grants and individual fellowships to ensure a continuing supply of well-trained scientists prepared to conduct cutting-edge research related to musculoskeletal, skin, and rheumatic diseases. The training program has been in existence since 1974. It was first funded at NIAMS in 1987, at which time it represented about 0.75% of NIAMS budget. Training grant and career award funding currently represents about 5.9% of the NIAMS budget. This compares to about 4.8% for all of NIH.

The specific grants and awards that were evaluated are the National Research Service Award (NRSA) institutional training grant (T32), NRSA individual research training grant (F32), and Mentored Career Development Awards (K01 and K08). While NIAMS uses other grant and award mechanisms, these awards were selected both because they represent a high proportion of the total dollars awarded, and because there is sufficient information available about recipients to assess their career progress over time.

The NIAMS evaluation had two phases. The first phase was a feasibility study, which was completed in October 2006. The second phase was this outcome evaluation. A working group of outside experts was convened to provide input into the feasibility study and the evaluation design, to help define success for trainees, and, based on the findings of the evaluation, to provide recommendations to the NIAMS Director on how the program might be improved in the future.

The outcome evaluation addressed two overarching questions: (1) Have the training programs helped to maintain the research pipeline of musculoskeletal, skin, and rheumatic disease researchers? and (2) Is the existing structure still appropriate to meet current training needs? The primary performance measure was the number of trainees that have developed successful research careers in fields relevant to NIAMS as defined by their ability to secure independent funding such as R01 grants, their publication history, their current employment, and their participation in activities such as professional organizations and conferences. Because multiple factors affect the success of a research career, multiple variables were examined, and success was viewed as a continuous variable rather than a binary determination.

The study used two approaches to data collection, quantitative and qualitative. The quantitative data collection provided input on the career outcomes of the trainees. The qualitative data collection consisted of two parts: (1) interviews with NIAMS Extramural Program (EP) program directors and grant review and management staff to gather their views on the training grant and career awards program; and (2) informal discussions between members of the working group and their colleagues in NIAMS-related fields.

The quantitative research was intended to address the first overarching question posed in the evaluation. Several detailed questions were grouped to address four aspects of becoming a successful researcher, namely:

Has the training award recipient continued in a research career after training?
If yes, has the trainee become a productive independent researcher?
Is the trainee currently working in a field relevant to the NIAMS mission?
Is there appropriate diversity in the training grant program?

The qualitative research addressed the second overarching question of the evaluation. It examined through staff interviews the strengths and weaknesses of the current program regarding whether it has been meeting the NIAMS goals of ensuring a continuing supply of well-trained scientists prepared to conduct cutting-edge research related to NIAMS mission areas. The interview questions sought to determine whether any of the four award types should be modified to help them more effectively assure that there are sufficient, qualified future researchers in areas of interest to NIAMS. In addition, working group members held informal discussions with colleagues to gather their views on the program.

The primary approach to determining the career outcomes of the grant and award recipients was to use public and NIH databases to gather information on a sample of recipients. Four cohorts were examined, one for each type of grant or award being studied . The T32 and F32 cohorts were chosen from trainees that received grants during 1993 or 1994. The K01 and K08 cohorts consisted of trainees that received awards during 1996 or 1997. These periods were chosen because they provided sufficient lag time between the start of training and the evaluation and to allow for career progression. That is, if a T32 or F32 recipient was in the postdoctoral stage when the training grant was received, 12-13 years should have been sufficient time for publications and other indicators of career choices and productivity to appear in public databases. For the career development awards, 10-11 years should have been sufficient time to see progress. In total, there were 379 recipients included in the sample.

1.2 Findings

1.2.1 Definition of success

As a first step in evaluating the success of the NIAMS training grant and career development program, the working group saw a need to establish a working definition for "success" from each of three perspectives: the individual trainee, the individual academic training program (e.g., an academic institution that receives a T32 grant), and the NIAMS training program overall. Each of these is described below.

Individual Trainee Success

For the purpose of this evaluation, the outcome for an individual trainee was considered to be a success if his/her career fell into one of the following two broad categories:

A career in which research is the primary focus; for example, full-time researchers in academia, industry, and government, as well as research administrators
A career in which research is a secondary focus; for example, educators in a research environment and clinicians that contribute to research in ways such as participation in studies led by others

Career paths of training or career award recipients that fell outside of these categories, while valuable, were considered unsuccessful within the scope of this evaluation, given that the goals of the training grant and career award programs are to develop researchers to meet future scientific needs of NIAMS. A precise cut off point for success on the career spectrum for each grant and award type was not determined ahead of time, however.

Training Program Success

The success of training programs at specific academic institutions (e.g., institutions that receive T32 awards) should be defined, at least in part, by the percent of recipients at each institution who go on to "successful" careers as defined for individual trainees above. However, at present, there is no evidence-based analysis or community consensus that can be cited to establish what percentage should be accepted as the definition of success. In the absence of such a "community standard", the working group judged that, in the current funding environment, a retention rate of >50% in research-oriented careers is a reasonable goal. The following additional characteristics were identified as important qualities of successful training programs:

Shows diversity in race, ethnicity, and gender of trainees
Fosters an environment conducive to interdisciplinary or multidisciplinary research, as appropriate
Promotes innovation and responsiveness to the current scientific environment
Creates or maintains an environment that supports career development, synergy between trainer and trainee, and scientific accomplishment

NIAMS Success

The overall NIAMS training program should be judged at least in part by the percentage of trainees who ultimately devote their careers to research. In addition to producing a pipeline of researchers that meet the criteria for individual success, the group identified the following factors that would indicate successful NIAMS administration of the training grant and career award programs. These factors included supporting training programs that yielded:

Scientific progress that promotes improved public health (prominently including, but not limited to, clinical/translational research)
Comprehensive and innovative programs in NIAMS' mission areas
Appropriate approaches to addressing current priorities and planning for future needs
Sufficient recruitment and retention of researchers
Diversity in institutional size; geographic distribution; and in race, ethnicity, and gender of trainees

The working group concurred with the approach of evaluating the different NIAMS programs independently, because they have different objectives and target audiences. Although the training sequence funded by NIH is often referred to as a pipeline, it is, in fact, more of a funnel. There are more trainees at the early stages of their research careers, such as in the T32 program, than there are in the later mentoring stages found in the K career award program. But even getting a K award is no guarantee of being able to secure an R01 grant. Therefore, it is to be expected that lower percentages of T32 recipients will eventually win an R01 grant than K award recipients, and not all K award recipients will successfully apply for an R01 grant.

1.2.2 Evaluation Findings

The data findings for trainees are summarized below by grant or career development award type. The detailed data are included in the body of the report in Section 7.0. Table 1 is a summary of some of the key findings for trainees in the study sample.

Table

1 Summary of Key Findings by Grant Type

Indicator/Grant Type	T32	F32	K01	K08
Current science-related career	75%	84%	100%	100%
Currently participate in research	54%	68%	100%	62%
Received R01 grant	17%	34%	83%	55%
Received other NIH funding	37%	59%	--	69%
Published during the past 6 years	55%	68%	100%	84%
Active in professional associations	58%	57%	50%	91%
Received professional award or honor	20%	21%	17%	59%
Male	58%	55%	*	51%
Female	42%	35%	*	45%
Average age	34	32	*	37

* K01 data were combined with K08 data due to the small size of the K01 cohort.

The data are briefly explained below:

T32 Grants:

The working group's definition of success for a T32 trainee was the broadest. This was generally in conformance with the views of most of the NIAMS EP staff that were interviewed. Although it is not expected that every T32 recipient will become an independent researcher with an R01 grant, every T32 recipient benefits from the exposure to scientific research methods and may apply the knowledge and experience gained in any number of related pursuits.

Overall, 75% of the T32 recipients have stayed in a science-related career, 54% have current job titles indicating active participation in research (e.g., PI, researcher, professor, instructor or assistant professor), and 55% published research in a NIAMS mission-related field during the last six years. Seventy-eight percent of the recipients published during the last 10 years, and 50% published during the past two years. T32 recipients were the primary authors in 45% of their 951 publications. Fifty-eight percent are active in at least one professional association, and 20% received at least one professional award or honor. Seventeen percent of the T32 recipients received R01 grants, and 37% received some sort of NIH grant after the T32.

There is limited geographic diversity in the institutions receiving the T32 grants. During 1993-1994, the institutions receiving T32 grants were located in 19 states. However, 10 states received 84% of the grants, and of these, four states received 56% of the grants ( California - 20, Pennsylvania - 13, Massachusetts - 12, New York - 11). This distribution reflects in large part the uneven geographic distribution of leading academic research centers in the United States . Although it was not possible to separate out the demographic information for the T32 cohort subsample, for all T32 recipients during 1993-1994, 58% were male and 42% were female. Self-identified race and ethnicity data for the trainees showed their makeup to be 48% White, 19% Asian, 2% Black or African American, 3% Hispanic (race or ethnicity), and 28% unknown. There were no self-identified American Indian/Native American or Pacific Islander trainees. The average age of the trainees was 34 years old.

Overall, the working group indicated that the T32 program was successful. The trainee outcomes are summarized below in Figures 1 and 2.

Figure 1 T32 Publishing Outcomes

Figure 2 T32 Professional Outcomes

F32 Grants:

Because the F32 grants are awarded to individuals, the working group had higher expectations of success for these trainees than for the T32 recipients. Overall, 34% of the F32 recipients went on to receive R01 grants, and 59% had received some sort of NIH research grant after their F32 award. Eighty-four percent of the F32 recipients have stayed in a science-related career, 68% have current job titles indicating active participation in research (e.g., PI, researcher, professor, instructor, or assistant professor), and 68% published research in a NIAMS mission-related field during the last six years. Eighty-seven percent of the recipients published during the last 10 years, and 59% published during the past two years. F32 recipients were primary authors in 33% of their 362 publications. Fifty-seven percent are active in at least one professional association, and 21% have received at least one professional award or honor.

There is limited geographic diversity in the institutions receiving the F32 grants. As noted above, this geographic limitation closely corresponds with the uneven distribution of major academic research centers in the U.S. During 1993-1994, the institutions receiving F32 grants were located in 20 states. However, four states received 50% of the grants ( California - 8, Massachusetts - 7, Texas - 5, Wisconsin - 3). The1993-1994 cohort of trainees were 55% male, 35% female, and 10% unknown. The trainees self-identified as 87% White, 3% Asian, and 1% Black or African American. None of the recipients self-identified as American Indian/Native American, Pacific Islander, or Hispanic. Ten percent were of unknown racial or ethnic origin. The average age of the trainees was 32 years old.

The working group felt that, overall, the F32 program was successful. The trainee outcomes are summarized below in Figures 3 and 4.

Figure 3 F32 Publishing Outcomes

Figure 4 F32 Professional Outcomes

K01 Awards:

The working group had the highest expectations for the K award cohorts. In fact, 5 out of 6 of the K01 recipients have received R01 grants. Overall, 100% of the K01 recipients have stayed in a science-related career, 100% have current job titles indicating active participation in research in a university setting (e.g., Assistant or Associate Professor), and 100% published research in a NIAMS mission-related field during the last six years. One hundred percent of the recipients published during the last 10 years, and 83% published during the past two years. K01 recipients were primary authors in 38% of their 79 publications. Fifty percent are active in at least one professional association, and 17% have received at least one professional award or honor.

Of the six K01s awarded, three were in Tennessee, one in Illinois, one in New Hampshire, and one in Arizona. Because this cohort was so small, the demographic information on award recipients has been combined with the K08 data. The outcomes are summarized below in Figures 5 and 6.

Figure 5 K01 Publishing Outcomes

Figure 6 K01 Professional Outcomes

K08 Awards:

Overall, 55% of the K08 recipients have received R01 grants, and 69% have received other types of NIH grants besides the K08. One hundred percent of the K08 recipients stayed in a science-related career, 62% have current job titles indicating active participation in research (e.g., PI, researcher, professor, instructor, or assistant professor), and 84% published research in a NIAMS mission-related field during the last six years. Ninety-eight percent of the recipients published during the last 10 years, and 85% published during the past two years. K08 recipients were primary authors in 40% of their 613 publications. Ninety-one percent are active in at least one professional association, and 59% have received at least one professional award or honor.

As with the other programs, there is limited geographic diversity in the institutions with K08 award recipients. During 1996-1997, the institutions receiving K08 grants were located in 20 states. However, five states received almost 50% of the grants ( Massachusetts - 11, New York - 5, California - 4, Pennsylvania - 4, Missouri - 4). The K08 and K01 recipients were 51% male, 45% female, and 4% unknown. They were self-identified as being 76% White, 10% Asian, 0% Black or African American, 2% American Indian/Native American, 2% Pacific Islander, 5% Hispanic, and 5% unknown. They had an average age of 37 years old.

The working group speculated that the success rate for K08 awards (100% retention in science-related careers) may actually be an indicator that K awards are not available in sufficient numbers, and that the need may substantially exceed the program size. Paradoxically, a somewhat lower success rate might provide a stronger indication that the program has reached the people it should. In addition, given that K08 applicants are typically at a stage of their careers where alternative sources of support cannot be sustained through a series of NIH application cycles, the lengthy cycle for resubmitting K awards is an area needing improvement. Addressing K award issues should be a priority due to the data illustrating that the K awards are successful in creating new investigators. The outcomes are summarized below in Figures 7 and 8.

Figure 7 K08 Publishing Outcomes

Figure 8 K08 Professional Outcomes

Based on information provided by Dr. Helen Lin of the NIAMS EP Scientific Review Office, there is a 38% overall success rate for K awardees in applying for R01 grants. This compares to a general success rate of 20%. Among a sample of 14 K01 awardees who have received R01 grants, four were awarded on the first attempt, five at the second submission, and five at the third submission. There are three submission dates for new K applications each year: February 12, June 12, and October 12. For each submission, it takes at least five months to receive the Summary Statement response from the reviewers, then the applicant can resubmit it in the next submission date. Thus, an application submitted on February 12 will get the Summary Statement back in July and then it can be resubmitted on November 12 (the resubmission date is one month later than the new submission date).

Based on information provided by Dr. Lin, of the 44 K08 awardees funded by NIAMS in FY 2000, 33 applied for R01 grants and 22 were successful (50%). As mentioned previously, the general success rate for R01 grants is 20%. Among the 22 R01 awardees, nine were successful on the first attempt, six were successful upon their second submission, and seven received the R01 grant with their third submission.

Qualitative Findings

The qualitative analysis consisted of interviews with the NIAMS EP program directors and grant management and review staff, as well as discussions between members of the working group and their colleagues around the country. This analysis yielded many observations about each of the grant and award programs being studied. These summarized observations are included in Section 8 of this report. In looking at the overall program, the interview participants identified the strengths of the overall program, namely: (1) it supports the next generation of researchers; (2) there is good balance between individual award and grant programs; and (3) because there are several training fund mechanisms available, they are able to reach a variety of people and fill various needs along the researcher's career path.

The interviewed participants also identified the following weaknesses of the overall program: (1) the programs need to focus on management training as well as the actual experiments - project management, budgeting, and staffing are important skills for Principal Investigators (PIs); (2) there should be greater emphasis on translational research in the application review process; (3) the program needs greater emphasis on mentoring; and (4) there is a need for more comparability in the application review process between/among study sections. In addition, the participants observed that many universities' expectations for tenure are not realistic - not everyone will get an R01 grant.

Participants suggested modifications to the overall program, which included: (1) more emphasis on interdisciplinary approaches when awarding grants; (2) linking training programs with Centers to get more synergy and cross-training; (3) conducting a separate examination of the NIAMS application review process; (4) providing help to institutions in order to harmonize programs so that all grants are maximized (e.g., if an institution supports recipients of several types of grants, there should be interaction between/among the programs); (5) focusing on good mentors and mentor training as part of grant oversight; and (6) looking to fill gaps in the research pipeline when awarding grants.

Some of the participants suggested that a useful definition of whether the program is successful would be to determine whether there are sufficient qualified applicants in fields where research is desired, rather than simply focusing on how many trainees eventually get R01 grants.

1.3 Limitations

Due to the retrospective nature of this evaluation and the limitations on the available data, the working group found that the study raised several questions that were outside the scope of the study but were of interest to the group. Most prominent among them were:

Whether a cause and effect relationship exists between the NIAMS training programs and the eventual success of individual trainees
The level of NIAMS performance compared to other NIH Institutes
Whether there were hidden deficiencies in any of NIAMS distinct mission areas (i.e., arthritis, skin, and musculoskeletal) that were not evident in this broad analysis
The attrition rate from research careers factoring in older cohorts, because data from the earlier phases of the awardees' careers (10-15 years) may overestimate success in retaining investigators over the long haul
The quality and impact of the contributions made by the trainees over the course of a career; that is the degree or quality of success in research as distinguished by major accomplishments such as membership in the National Academy of Sciences and major research awards (that may come later in careers), as compared to less substantial contributions or awards (that may be received earlier in researchers' careers).

1.4 Working Group Recommendations

Despite the limitations cited above, the committee was impressed by the success of all the NIAMS training programs as measured by the criteria outlined in section 1.2.1. The group believes that each of the training grant and career development programs has served an important purpose and should be continued. In addition, the committee recommends the following:

R1: Establish a structured data collection mechanism to support ongoing evaluation of training grant effectiveness by type of grant.

The retrospective nature of this evaluation highlighted the need to design and implement prospective mechanisms that can assess the success of individual trainees over the course of their research careers, as well as the success of institutional training programs and each individual NIAMS training mechanism. This includes establishing control groups such as peers who did not receive training grant support from NIAMS, or trainees from other NIH institutes or private foundations. NIAMS should determine what data are needed and make necessary revisions to the applications to obtain these data prospectively. Even more important, NIAMS should implement appropriate follow-up evaluations on an on-going basis to assess the precise nature, quality, and duration of each trainee's career. The information then should be captured in a database and used to continue to evaluate the program.

R2: Acknowledge the economic aspects of research by providing more flexibility on the percent effort required for K awards to accommodate clinical responsibilities and other personal and professional circumstances and by lifting restrictions that limit other sources of funding.

The current K award program generally requires that trainees spend 75% of their time on research. This can create difficulties for M.D.s who may also be expected to maintain a clinical practice and teach. The percent of time required to be devoted to research, as well as limitations on other sources of funding that could support the trainee, is believed to be a significant barrier to entry into a research career for many M.D.s. By adding flexibility in these areas to the K award program, NIAMS may be able to attract physicians and fill current gaps in the research pipeline. NIAMS has already increased support to M.D. K99 recipients up to $75,000 per year for the first two years and has lowered the research time requirement for surgeons who receive K08s to 50% effort.

R3: Avoid imposing a time limit from completion of degree on applications. Maintain flexibility and discretion of the peer review board to reward outstanding candidates.

Data provided to the evaluation showed that the process of applying for grants can be long. Many applicants have to apply multiple times until they are successful. In addition, people may slow down their timeline for career development post completion of their degree due to factors such as starting a family. There didn't seem to be much benefit in imposing a time limit on applications, because these situations are already factored into the review process.

R4: Build on current success - illustrated by initial data collection - of the training grant and career development award program to leverage the recent increase of participants in NIAMS mission-related programs by increasing funding for NIAMS training grant mechanisms. The pipeline of researchers cannot be expanded unless the number of awards and the amount of funding is also increased.

The data demonstrated that trainees are, by and large, accomplishing the goals of the training grant and career award program. NIAMS should look at selectively increasing the funding for awards. This would include funding to enhance the T32 grants to encourage collaboration between programs and interdepartmental work, support for developing the mentor-trainee relationship, adding additional support for the F32 grants, and increasing the funding levels for K awards . The working group speculated that the success rate for K08 awards (100% retention in science-related careers) may actually be an indicator that K awards are not available in sufficient numbers and that the need may substantially exceed the program size. Paradoxically, a somewhat lower success rate might provide a stronger indication that the program has reached the people it should. In addition, given that K08 applicants are typically at a stage of their careers where alternative sources of support cannot be sustained through a series of NIH application cycles, the lengthy cycle for resubmitting K awards is an area needing improvement. Addressing K award issues should be considered a priority due to the data illustrating that the K awards are successful in creating new investigators.

In making this recommendation, the working group is keenly aware that budget increases in one area often require cuts in other areas. Accordingly, this recommendation will have to be considered in the context of the overall NIAMS priorities. While it is beyond the scope of this group to address these broad priorities, the members of the working group feel that they would be remiss if this report did not highlight the success of the training programs and the likelihood that they could be even more successful if they were funded at a higher level. However, it should be emphasized that this recommendation is not meant to imply that R01 support should be reduced in favor of training support (see recommendation 6 below).

R5: Consider integrating a new component into NIAMS institutional training grant strategy that would address the related dilemmas of prolonged training followed by multiple application cycles in pursuit of a K award, which were seen as major deterrents to a career in science.

The members of the working group shared a common concern about the length of time required for trainees to establish themselves, and the frequent necessity to endure multiple application cycles before achieving K level funding. The adverse impact of these factors on retention of promising young investigators is regarded as a significant problem that warrants attention. However, the working group did not reach consensus on how best to address this problem. Three ideas each received some degree of support:

Some committee members favored implementation of a new mechanism in which selected trainees with particularly high potential could be identified early in their training by the institution rather than by NIAMS and provided with K level support (in dollars and duration).
Some members of the committee favored implementation of a bridge-type award that would support trainees at a K level for 2-3 years after fellowship training while they sought a K award. One way of implementing this idea might be by the addition of junior faculty positions to established T32 programs so that selected trainees could be provided with financial support after completing their fellowship training. The support should be equivalent to a K award but perhaps of shorter duration, with the goal to provide a bridge while the individuals sought a K award.
Some members of the committee favored using whatever funds might be devoted to the two proposed mechanisms above to expand funding for the existing K programs, and thus reduce the likelihood that an applicant might have to survive several application cycles before succeeding.

R6: Increase NIAMS budget for R01 grants so that there are more opportunities for trainees to conduct independent research at the end of the pipeline.

The most significant impediment to attracting and retaining qualified individuals for careers in NIAMS-related fields is the (accurate) perception that this is a high-risk career path. All NIAMS-supported trainees eventually face the increasingly daunting challenge of achieving and maintaining independent R01 support. This problem is the single most important reason for departure from research careers in favor of other options (e.g., clinical practice), as evidenced by surveys such as those conducted by the American College of Rheumatology. Given this reality, the most important measure that can be taken to allow trainees to achieve successful research careers (and, by so doing, validate the success of NIAMS training programs) is to insure that there is a reasonable likelihood of support for them at the other end.

R7: Centralize training information to make information on different mechanisms more accessible to potential applicants. Encourage collaborative interaction with professional and constituent organizations to develop a robust complementary portfolio of training funding.

One concern that was expressed by evaluation participants at NIAMS was that applicants did not necessarily have complete information about all the possible sources of funding that might be available to them. This is particularly important for those eligible for the career development awards that may need additional income beyond that provided by the NIAMS grant. In addition, many professional associations and constituent organizations have research arms that can help fund promising researchers in their field of interest. NIAMS could work closely with these organizations to strengthen the diversity of offerings and help provide a strong, visible network of support for researchers early in their careers.

R8:Structure the criteria for success in grant review to encourage and reward integrated and interdepartmental approaches, foster innovation, and support interdisciplinary mentorship in applications. Reinforce the value of grant writing and management in program curriculum.

The qualitative portion of the evaluation identified several areas where NIAMS program managers thought the overall quality of the training grant and career award programs could be improved. These included more integration between different departments at institutions in order to reflect the changing research environment, which continues to be more interdisciplinary. It is important to train researchers to be able to work effectively as research methods continue to evolve, and team efforts increase in importance. Interdisciplinary teams also require more skills to manage. Teaching the trainees project management and grant writing skills while they are early in their careers will contribute to their ability to become effective, independent PIs later. By rewarding these approaches through grant funding, NIAMS can influence the behavior of the institutions and individuals applying for grants. The study sections and NIAMS staff can play an important role in giving feedback to applicants on how the criteria are being applied.

R9: Reinforce the value of mentorship by providing a range of opportunities (e.g., annual meetings at NIAMS, web-based modules, etc.) that support training of mentors as well as trainees and that foster an environment of collaboration and support for mentors and those being mentored.

The NIAMS managers that participated in the qualitative interviews and the working group were in agreement that mentors play a key role in developing successful researchers. Supporting the development of good mentors is an important investment that NIAMS should make in the training grant program. In addition to helping trainees find good mentors, NIAMS can also proactively reach out to mentors and trainees to help them understand and develop their relationship. Mentors and those being mentored need to be trained in mentorship, and NIAMS is in a position to play an important role in creating an environment where this can take place. NIAMS can also work with professional associations and organizations to encourage them to be partners in promoting high functioning mentors who can make a significant difference in the training environment.

R10: Work with other NIH Institutes and private foundations to insure that there is a comprehensive and complementary portfolio of funding mechanisms for trainees.

NIAMS is not the sole source of training support for young investigators who are interested in NIAMS-related research areas. At present, the universe of training grant mechanisms is fragmented among several NIH institutes and numerous other potential funding sources. It would be in the best interest of trainees if the various agencies that support training in these areas coordinated their activities to insure maximum efficiency and appropriate balance.

2.0 Background

2.1 Overview of Program

The program being evaluated is the NIAMS extramural research training and career development award program. Like other NIH training and career development grants and awards programs, the NIAMS program is intended to help ensure that a diverse and highly trained workforce is available to assume leadership roles related to biomedical and behavioral research. NIAMS' overall objective is to use a combination of institutional training grants and individual fellowships to ensure a continuing supply of well-trained scientists are prepared to conduct cutting-edge research related to musculoskeletal, skin, and rheumatic diseases. The training program has been in existence since 1974. It was first funded at NIAMS in 1987, at which time it represented 0.75% of the NIAMS budget. Training grant and career award funding currently represents about 5.9% of the NIAMS' budget. This compares to about 4.8% overall across NIH.

The specific grants and awards that were evaluated are the National Research Service Award (NRSA) postdoctoral institutional training grant (T32), NRSA postdoctoral individual research training grant (F32), and Mentored Career Development Awards (K01 and K08). While NIAMS uses other grant and awards mechanisms, these awards were selected both because they represent a high proportion of the total dollars awarded, and because there is sufficient information available about recipients to assess their career progress over time. The table below shows historical funding for these programs during the periods being evaluated. Note that the K01 program was not funded until 1996.

Table 2 Training Grant Funding by Fiscal Year ($ in m)

	1993	1994	1995	1996	1997	1998	1999	2000	2001	2002	2003	2004	2005	2006
T32	5.5	5.9	6.1	6.3	6.2	5.7	7.4	7.1	8.3	10.3	10.9	11.1	11.7	11.4
F32	0.9	1.0	1.1	1.2	1.4	1.3	1.5	1.8	2.1	2.3	1.8	1.8	1.8	2.8
K01	--	--	--	0.1	0.4	0.8	1.3	2.1	3.4	3.6	4.3	4.3	4.3	4.7
K08	3.0	3.4	3.9	4.3	4.5	4.0	4.8	5.2	4.5	4.5	4.1	4.0	3.8	3.9
TOTAL	9.4	10.3	11.1	11.9	12.5	11.8	15.0	16.2	18.3	20.7	21.1	21.2	21.6	22.8

Below is a description of each of these awards.

T32: The postdoctoral T32 is designed to provide a broad training experience for future researchers just beginning their postdoctoral work. It exposes the research trainee to the basic approaches for conducting scientific research under the guidance of an experienced investigator who will help them start down the path to becoming an independent researcher. The T32s are awarded to institutions, which then select the trainees.

F32: The postdoctoral F32 is designed to support new researchers with high potential for becoming independent investigators. More competitive than a T32, the F32 is an NIH-reviewed award that is given directly to the trainee and moves the postdoctoral fellow farther along the path towards independence.

Does not include all NIAMS training grants and awards.

K01: The K01 or Mentored Research Scientist Development Award provides support for a sustained period of protected time (three, four, or five years) for intensive research career development under the guidance of an experienced mentor, or sponsor, in the biomedical or clinical sciences leading to research independence. The expectation is that through this sustained period of research, career development, and training, awardees will launch independent research careers and become competitive for new research project grant (R01) funding.

K08: The K08 or Mentored Clinical Scientist Development Award is designed to support the development of outstanding clinician research scientists. This mechanism provides specialized study for individuals with a health professional doctoral degree committed to a career in laboratory or field-based research.

Table 3 is a summary of the characteristics of the grants and awards studied.

Table 3 Summary of Characteristic of Grants and Awards Studied

Mechanism	(T32) NRSA Institutional Training Grants Postdoctoral Training	(F32) National Research Service Awards Postdoctoral Individual	(K01) Mentored Research Scientist Development Investigator Awards	(K08) Mentored Clinical Scientist Development Investigator Awards
Purpose	To enable institutions to make NRSAs for postdoctoral research training to individuals selected by them in fields of arthritis, muscle, bone, musculoskeletal, and/or skin diseases.	To provide postdoctoral research training to individuals to broaden their scientific background and extend their potential for research in arthritis, muscle, bone, musculoskeletal and/or skin diseases.	To support intensive, supervised career development experience in one of the biomedical, behavioral, or clinical sciences leading to research independence.	To support clinicians who need an intensive period of mentored research experience.
Level	M.D. or Ph.D. or equivalent.	M.D. or Ph.D. or equivalent.	Research or health-professional Ph.D. and postdoctoral research experience.	M.D. or Ph.D. or equivalent; clinician.
Duration	5-year institutional grant (renewable). A trainee may have up to 3 years of postdoctoral support.	Up to 3 years.	Up to 5 years.	Up to 5 years.
Provisions	Stipend $28,260 to $44,412 per year. Tuition support.	Stipend $28,260 to $44,412 per year.	Support is provided for salary up to $75,000, fringe benefits, and other research expenses up to $20,000.	Support is provided for salary up to $75,000, fringe benefits, and other research expenses up to $20,000.
Unique Characteristics	Must be U.S. citizen or permanent resident. Recipients selected by institution, not NIAMS.	Must be U.S. citizen or permanent resident.	Must be U.S. citizen or permanent resident. Development in a new area of research. Salary determined by the sponsoring Institute. Can be supplemented by the institution to match salaries of those with similar preparation in the setting. 75% research effort required.	Must be U.S. citizen or permanent resident. Development of the independent clinical research scientist. 75% research effort required except for surgeons who have a 50% requirement and reduced salary of $50,000.

2.2 Need for an Outcome Evaluation

NIAMS is committed to offering training support for current and future basic and clinical researchers focused on core components of its mission. The outcome evaluation was designed to examine outputs from NIAMS-supported research training and career development award programs to determine whether the program has been reaching its overall goal of ensuring that a diverse and highly trained workforce is available to assume leadership roles related to the Nation's biomedical and behavioral research agenda. In addition the evaluation examined whether NIAMS' overall objective of using a combination of institutional training grants and individual fellowships to ensure a continuing supply of well-trained scientists prepared to conduct cutting-edge research related to NIAMS mission areas has been met. Specifically, the outcome evaluation addressed two broad questions: (1) Have the training programs helped to maintain the research pipeline of musculoskeletal, skin, and rheumatic disease researchers?; and (2) Is the existing structure still appropriate to meet current training needs?

This evaluation is an important element of NIAMS strong support for the trans-NIH goal of developing a cadre of interdisciplinary research scientists. An important role of the working group of outside experts that was convened to provide input into the evaluation's feasibility and to assess the evaluation's results was to define what successful outcomes are for trainees. Currently, there is no clear NIH-wide standard for success. Other studies that may be conducted of the NIH training grant programs will provide an outside benchmark against which the success of the same training mechanisms at NIAMS may be compared.

2.3 Outcome Evaluation Approach

The evaluation had two phases. The first phase was a feasibility study, which was completed in October 2006. The second phase was this outcome evaluation. A working group of outside experts provided input into the feasibility study, examined the collected data and analyses related to the training program outcomes, and made recommendations for future directions of the program, which are included in this report. The primary information gathering vehicles for the evaluation were collection and analysis of data on the career outcomes to date of the trainees and conducting NIAMS EP staff interviews. Other information was provided through the informal interactions of the working group members with their peers and studies conducted by NIAMS EP staff.

2.4 Role of the Working Group of Outside Experts

The working group was convened by the NIAMS Director to provide methodological advice about the evaluation design, to review and analyze the quantitative and qualitative data collected as part of the evaluation, and, based on its findings and on the expertise of its members, to develop a set of recommendations for the future of the training grant and career awards programs. The working group consisted of both outside experts and representatives of NIH. The members were selected by the NIAMS Director because of their knowledge of the NIAMS program areas and familiarity with the training programs, either as a previous recipient, mentor, and/or program director.

On July 25, 2006, the working group met at NIAMS to assess the feasibility of the outcome evaluation and to provide input into the evaluation design for the four selected mechanisms. The working group's findings and recommendations regarding feasibility and study design can be found in the Training Grant and Career Award Program Evaluation Working Group Feasibility Report , October, 2006.

After the data were collected and analyzed, the working group reviewed the data and findings and developed recommendations to NIAMS for areas of improvement. Appendix 1 lists the working group members. Appendix 2 lists the NIAMS EP staff that participated in the evaluation effort.

2.5 Definition of Success

Individual Trainee Success

For the purpose of this evaluation, the outcome for an individual trainee was considered to be a success if his/her career fell into one of the following two broad categories:

A career in which research is the primary focus; for example, full-time researchers in academia, industry, and government, as well as research administrators
A career in which research is a secondary focus; for example, educators in a research environment and clinicians that contribute to research in ways such as participation in studies led by others

Training Program Success

Shows diversity in race, ethnicity, and gender of trainees
Fosters an environment conducive to interdisciplinary or multidisciplinary research, as appropriate
Promotes innovation and responsiveness to the current scientific environment
Creates or maintains an environment that supports career development, synergy between trainer and trainee, and scientific accomplishment

NIAMS Success

Scientific progress that promotes improved public health (prominently including, but not limited to, clinical/translational research)
Comprehensive and innovative programs in NIAMS' mission areas
Appropriate approaches to addressing current priorities and planning for future needs
Sufficient recruitment and retention of researchers
Diversity in institutional size; geographic distribution; and in race, ethnicity, and gender of trainees

3.0 Conceptual Framework

This conceptual framework shows the inputs into the research training process and the external program factors affecting the process, as well as the desired activities of the trainees and the program goals. The evaluation did not look at the effectiveness of the process by evaluating program activities, nor did it examine external factors. The evaluation did examine the extent to which trainees are participating in the listed activities in order to determine whether the goals of the program are being achieved. Resources devoted to the program and population characteristics were also included, although they were not determinants of whether goals are being achieved. Figure 9 below illustrates the conceptual framework for short, medium, and long term NIAMS goals for the training programs.

Figure 9 Conceptual Framework

4.0 Study Research Questions

The study questions were designed to answer the overarching questions posed by the evaluation: (1) Have the training programs helped to maintain the research pipeline of musculoskeletal, skin, and rheumatic disease researchers?; and (2) Is the existing structure still appropriate to meet current training needs? There are both quantitative and qualitative questions as described below.

The primary performance measure was the number of trainees that have successful research careers in fields relevant to NIAMS as defined by their ability to secure independent funding such as R01 grants, their publication history, their current employment, and their participation in professional activities such as professional organizations and conferences. Because multiple factors affect the success of a research career, multiple variables were examined, and success was viewed as a continuous variable rather than a binary determination.

The quantitative questions were intended to address the first overarching question posed in the evaluation, and were grouped to address four aspects of becoming a successful researcher, namely:

Has the training award recipient continued in a research career after training?
If yes, has the trainee become a productive independent researcher?
Is the trainee currently working in a field relevant to the NIAMS mission?
Is there appropriate diversity in the training grant program?

Questions 1-3 addressed the overall goal of ensuring that a diverse and highly trained workforce is available to assume leadership roles related to the Nation's biomedical and behavioral research agenda, as well as NIAMS goal to use a combination of institutional training grants and individual fellowships to ensure a continuing supply of well-trained scientists prepared to conduct cutting-edge research related to NIAMS mission areas. Question 4 addressed the diversity element of the overall program goal.

The qualitative research addressed the second overarching question of the evaluation, examining through staff interviews the strengths and weaknesses of the current program regarding whether it has been meeting the NIAMS goals of ensuring a continuing supply of well-trained scientists prepared to conduct cutting-edge research related to NIAMS mission areas. The interview questions sought to determine whether any of the four award types should be modified to help them more effectively assure that there are sufficient, qualified future researchers in areas of interest to NIAMS. The following sections are the detailed questions within each of the four groups of quantitative areas.

4.1 Quantitative

2a. Independent research funding

2b. Research publications

2c. Professional activities and recognition

- Has the training award recipient continued in a research career after training?
- For each training award type, what proportion of trainees is conducting independent research, and what are their career choices, professional fields of study, and settings?
- For those trainees involved in research, is the trainee an individual whose career is primarily focused on research, including full-time researchers in academia, industry, and government as well as research administrators; or is the trainee an individual whose research is a secondary focus in their career, including educators in a research environment and clinicians who contribute to research led by others?
- Has the trainee become a productive independent researcher?
- For each training award type, what proportion of trainees has received independent research funding (e.g., private or public)?
- For what period of time was funding received?
- For the funding received, what was the field of study?
- What is the proportion of trainees who are participants in research grants but are not the principal investigator?
- What proportion of trainees has published in academic and peer-reviewed journals?
- Of those who have published, how often were they the primary author and how often a contributor?
- What are the dates of publication?
- What proportion of trainees is active in professional associations in their field?
- What proportion of recipients of NIH training support are also recipients of major honors for research achievement (e.g., from professional societies) in NIAMS areas of responsibility?
- Is the independent researcher working in a field relevant to the NIAMS mission?
- Do NIAMS-supported trainees continue to conduct NIAMS-related research throughout the course of their careers?
- Is there institutional diversity in the training grant program?
- What is the geographic location of the institution receiving the award?
- What is the size of the institution receiving the award?
- What are the demographic characteristics of the trainees (e.g., gender, race, or ethnicity)?

4.2 Qualitative

The following research questions were addressed through more qualitative means, such as interviews with the NIAMS EP staff and working group discussions with colleagues:

What are the strengths of the current program?
What are the weaknesses of the current program?
What are potential barriers to success?
If the current structure of the training programs is deemed to be successful in developing and maintaining the research pipeline, are there modifications that can be made to enhance the effectiveness of the training programs offered at NIAMS?
If not, could one or more of the training programs be significantly modified to meet the needs and opportunities within the current research environment?

A copy of the staff interview questionnaire is presented in Appendix 3 .

5.0 Study Methodology

The study used two approaches for data collection: quantitative and qualitative. The quantitative data collection provided input on the career outcomes of the trainees. The qualitative data collection consisted of interviews with NIAMS EP program directors and grant review and management staff. These two data collection methods are described in the following sections.

5.1 Quantitative

Approach

The primary approach to determining the career outcomes of the grant and award recipients was to use public and NIH databases to gather information on a sample of recipients. Table 4 shows the four cohorts that were examined and the number of recipients in each cohort . These periods were chosen because they provided sufficient lag time between the start of training and the evaluation to allow career progression to have occurred. That is, if a recipient was in the postdoctoral stage when the training grant was received, 12-13 years should have been sufficient time for publications and other indicators of career choices and productivity to appear in public databases. For the career development awards, 10-11 years should be sufficient to see progress.

Table 4 Grant Cohorts

Grant Type	Year Began Training	No. of Recipients
T32 Postdoctoral	1993 -1994	271
F32 Postdoctoral	1993 -1994	44
K01 Career Development	1996 -1997	6
K08 Career Development	1996 -1997	58
Total Recipients		379

Table 5 shows the total number of T32 and F32 training grants and K01 and K08 career development awards given out by NIAMS during the1990-2006 period. Note that the first K01 grants were awarded during 1996. Because of this, there were fewer of these grants given out during this 16-year period. In addition, as Table 2 shows, the 1996-1997 cohort for the K01 awards was considerably smaller than that for the K08 awards. The number of K01 awards given out each year has increased since 1996, and the program is currently comparable in size to the K08 awards.

Table 5 Total Grants and Awards at NIAMS

	T32	F32	K01	K08
TOTAL	979	715	268	636

Key Variables

The most important variables for which data will be collected are categorized and listed below. These refer specifically to NIAMS-related activities.

Program resources

Funding available for training grant and career awards (quantitative).

Population characteristics

The population being studied for career outcomes is divided into cohorts based on types of grants received, including subject of grant and institution and the year in which the award was received (quantitative).

Program activities

Areas in need of more interdisciplinary training; areas where training could be integrated with other relevant scientific activities that serve as fertile ground for training; evaluative components that measure success of trainees that should be incorporated into the training grant and career award program (qualitative).

Program goals and performance measures

Program goal: to help ensure that a diverse and highly trained workforce is available to assume leadership roles related to the Nation's biomedical and behavioral research agenda.
- Performance measure:Number of trainees that have published independent research papers (outcome measure)
- Performance measure: Number of trainees that qualify for additional NIH grants such as moving from a T grant to an F grant to a K award or R01 grant (outcome measure)
- Performance measure: Number of trainees that participate in professional activities such as conferences and NIAMS committees (outcome measure)
Program goal: To use a combination of institutional training grants and individual fellowships to ensure a continuing supply of well-trained scientists prepared to conduct cutting-edge research related to NIAMS mission areas.
- Performance measure: Number of trainees that have been awarded independent NIH grants such as the R01 or R03 (outcome measure) to conduct research of interest to NIAMS mission area.
- Performance measure: Leadership in scientific societies, academic appointments and tenure status, publications in peer reviewed journals, and independent research (outcome measure) in areas of interest to NIAMS.

Other variables of interest

Organization where trainee/researcher is or has been employed
Professional positions/environments
Organizational sector (e.g., private, public, non-profit)
Number of papers/publications the trainee/researcher has published, including dates and titles and where published
Type of publications
Whether trainee/researcher is the primary or secondary author
Whether trainee/researcher has appeared at conferences as a speaker, organizer, or session chair
Trainees/researchers active membership(s) in professional associations
Grant awards

Areas in which there was interest in but the data were not consistent or were difficult to collect on individual trainees included:

Grant participation (e.g., co-investigator)
Discovery (advancing the field
Essential contributions to research enterprise (through teaching and/or support for clinical research)
Timely career development
Duration of career (drop out rate)
Honors/societies/editorial leadership

Data Sources

The NIAMS database of records of grant applications and awards was used to pull together the sample file for each cohort of trainees whose career outcomes were studied. The Pub Med database was used to examine the publication history of the trainees. The NIH IMPAC II database, as accessed through the Training Activities (TA) database, was used to examine the NIH grant history of trainees. Generic web searches of publicly available data using Google and other search engines were used to gather information about trainee professional appointments and other related activities. Membership directories for professional associations were searched for training grant recipient participation.

Sampling

In order to reduce data collection time and costs, sampling methods were used to select an appropriate number of trainees in each cohort. Sample sizes for the evaluation were derived under the following assumptions:

The population sizes for the cohorts were the total number of recipients of each grant or award type for the years selected for the sample (see Table 5 below).
The cohorts for the F32, K01, and K08 grants were sufficiently small enough that they were not subsampled. Rather, data were collected on the entire cohort.
The T32 recipients were a sufficiently large enough group that the cohort was subsampled. The measure of interest was the proportion of trainees unsuccessful (successful) in the 1993-1994 cohort. The expected proportion of unsuccessful (successful) was 40% (60%).
Desired precision for the estimated proportion successful (unsuccessful) was + 8 percentage points, at a 95% confidence level.
The smallest groupings measured were the grant or award type. That is, the recipients were not divided or analyzed by any additional demographic characteristics other than having been a recipient of a particular grant or award.
Inability to locate information on a trainee was assumed not to be correlated with success level (i.e., inability to locate may be considered random due to errors in the databases, name changes over time, employment in the private sector, or other factors).

Given the small population sizes, the sample variance used in the sample size derivation formula took into account the finite population factor (fpc). Sample size for the T32 cohort shown below was derived accounting for some number of recipients that would not be found, called here the response rate or RoR, with the impact of RoR on the resulting precision provided.

Sample size derivation was based upon the sample variance, expressed as

where

sample size

population size

response rate (percent found)

The resultant sample size derivation formula was then

where

desired precision for estimated proportion unsuccessful (successful)

t-value associated with desired confidence level

The sample sizes are provided in the table below.

A total sample of approximately 217 trainees was included in the study. Table 6 shows the total number of recipients in each cohort and the sample size that was selected.

Table 6 Grant Recipient Cohorts Sample Size

TYPE	COHORT	N	p	d	t	RoR	n
T32 postdoctoral	1993-94	271	0.40	0.08	1.96	.88	109**
F32 postdoctoral	1993-94	44	*	*	*	*	44***
K01 career development	1996-97	6	*	*	*	*	6
K08 career development	1996-97	58	*	*	*	*	58
TOTAL		379					217

* Due to the small number of F32, K01, and K08 recipients, 100% of the cohort was included in the sample.

** Four recipients that were randomly selected as a member of the T32 subsample were later discovered to be part of the K08 cohort as well. The recipients are included in both groups.

*** One F32 recipient was also in the K01 cohort. The recipient is included in both groups.

Data Analysis

Descriptive statistics were used to answer the study questions on career outcomes, research productivity, and research diversity of grant recipients. The data were compiled in Access.

5.2 Qualitative

The staff interview questionnaire was developed in consultation with the Deputy Director of the NIAMS EP and the Deputy Director of the NIAMS Office of Science Policy and Planning, with input from the working group. A copy of the questionnaire is presented in Appendix 3 .

Qualitative Variables

Program activities

Areas in need of more interdisciplinary training
Areas where training could be integrated with other relevant scientific activities that serve as fertile ground for training
Evaluative components that measure success of trainees that should be incorporated into the training grant and career award program
NIAMS-based activities to encourage new researchers
Review process and criteria for grant applications
Strengths and weaknesses of the individual grants and awards
Strength and weaknesses of the program as a whole
Definition of a successful program
Barriers to success

External factors

Role of the institutional programs supporting the trainees
Quality of the mentors supporting the trainees
Availability of funding for R01 grants
Availability of institutional funds to support researchers

Data Sources

NIAMS EP program directors and grant review and management staff (referred to as NIAMS EP staff) provided information about interdisciplinary issues, the changes in science, areas where training could be integrated with other activities, and other evaluative component variables.

Data Collection Instrument

A new data collection instrument was developed to conduct the staff interviews. The questionnaire was administered by the outside consultant in individual face-to-face interviews. The interviews took place on site in the interviewees' offices or in a private conference room.

Data Analysis

Qualitative analysis was used to examine data from the staff interviews. In order to protect the identity of individual respondents, the responses were coded and organized by topic.

Data Preparation

The data for each trainee were originally logged into Microsoft Word documents and into spreadsheets - one for each type of training assistance. The data were then aggregated into five tables based on trainee, with distinct searchable data. The tables were put into a Microsoft Access database, and a series of simple queries provided data for cross tabs, which were exported back to Excel spreadsheets for formatting.

6.0 Limitations

The evaluation was not designed to establish causality for future success in achieving independent researcher status. Rather, it was designed to retroactively track the career outcomes of trainees who had received funding support 10-15 years ago. Because this was a retroactive study, there was no control group. As a result, the evaluation did not determine whether the researchers would have had the same career outcomes absent the training grants. However, because the study was able to determine the career outcomes for the recipients, it will help formulate future studies that may be designed prospectively and include control groups.

The evaluation approach was limited by the lack of comprehensive data on each grant or award recipient. More comprehensive data could have been collected by expanding the group of stakeholders being interviewed to include former trainees, program directors at institutions receiving grants, PIs, and others. However, this would have added significantly to the cost and complexity of the study design, including the need to get OMB and IRB approvals. For the purposes of this evaluation, however, the variables identified provide sufficient information to get a sense of how successful the trainees have been.

Regarding validity measures, as mentioned earlier, there are no standard measures of success that currently exist for the training programs, so it was not possible to compare NIAMS outcomes with other benchmarks. Therefore, generalizations about the results of the evaluation beyond NIAMS will need to be made with caution. In addition, while the feasibility study considered adding additional cohorts, it was determined that the cohorts should be from earlier groups of grant recipients, dating from the 1980s, because of the time required to develop a research career. Unfortunately, accurate and consistent data from the 1980s are not available electronically, which made this approach infeasible for the study timeframe.

Post-data collection, a number of limitations with the data surfaced. These are described below:

Information on the career outcomes for twenty T32 and four F32 recipients could not be located. This does not necessarily mean that they didn't have research careers. For example, some of the trainees may have gotten married or divorced during the past 10-14 years resulting in a name change. In those instances, they may not have been located by the data search, which was name based, unless they continued to get grants from NIH, rather than other sources. Likewise, trainees that went on to research careers in the private sector and have not published also may not have shown up in the data search.
It wasn't possible to consistently determine whether each trainee's career was primarily focused on research, including full-time researchers in academia, industry, and government, as well as research administrators; or whether research was a secondary focus in the trainee's career, including educators in a research environment and clinicians that contribute to research by others. While this was evident for some of the trainees, for others it was less so.
The NIH R01 grant information database has an unknown number of data entry errors in it. These appear to be small in number.
It is difficult to identify when someone is a co-investigator on an NIH grant. The NIH electronic files include records for all applications for NIH research grants and contracts. However, the electronic records only include information on the principal investigator who submitted the application, and individuals who are co-principal investigators are not listed.
Regarding the qualitative data, several of the interview participants did not have extensive experience with all grant and award types being studied and were unable to provide comments related to all items. As a result, while some of the responses were quite detailed, others were quite short. The responses were categorized and summarized to capture the most relevant information.

Whether a cause and effect relationship exists between the NIAMS training programs and the eventual success of individual trainees
The level of NIAMS performance compared to other NIH Institutes
Whether there were hidden deficiencies in any of NIAMS distinct mission areas (i.e., arthritis, skin, and musculoskeletal) that were not evident in this broad analysis
The attrition rate from research careers factoring in older cohorts, because data from the earlier phases of the awardees' careers (10-15 years) may overestimate success in retaining investigators over the long haul
The quality and impact of the contributions made by the trainees over the course of a career; that is the degree or quality of success in research as distinguished by major accomplishments such as membership in the National Academy of Sciences and major research awards (that may come later in careers), as compared to less substantial contributions or awards (that may be received earlier in researchers' careers)

7.0 Quantitative Findings

7.1 Overview

The data are summarized below by grant or career development award type. The detailed data are provided in subsequent sections organized by research questions. The data are presented by grant type, reflecting that each grant or award type was evaluated on a stand alone basis. They are not considered comparable to each other regarding the expectations for successful outcomes for trainees, because the recipient groups are at different levels of progress in their careers.

T32 Grants:

Overall, 75% of the T32 recipients stayed in a science-related career, 54% have current job titles indicating active participation in research (e.g., PI, researcher, professor, instructor, or assistant professor), and 55% published research in a NIAMS mission-related field during the last 6 years. Seventeen percent of the T32 recipients have received R01 grants, and 37% have received some sort of NIH grant after the T32. Seventy-eight percent of the recipients published during the last 10 years, and 50% published during the past two years. T32 recipients were primary authors in 45% of their 951 publications. Fifty-eight percent are active in at least one professional association, and 20% have received at least one professional award or honor.

There is limited diversity in the institutions receiving the T32 grants. During 1993-1994, the institutions receiving T32 grants were located in 19 states. However, of these, 10 states received 84% of the grants, and four states received 56% of the grants ( California -20, Pennsylvania - 13, Massachusetts - 12, New York - 11).

Although it was not possible to separate out the demographic information for the T32 subsample, overall for T32 recipients during 1993-1994, 58% were male and 42% were female. Self-identified race and ethnicity data for the trainees showed their makeup to be 48% White, 19% Asian, 2% Black or African American, 3% Hispanic (ethnic or racial), and 28% unknown. There were no self-identified American Indian/Native American or Pacific Islander trainees. The average age of the trainees was 34 years old. Figures 10 and 11 show the summarized outcome data for the T32 trainees in the sample.

Figure 10 T32 Publishing Outcome Summary

Figure 11 T32 Professional Outcome Summary

F32 Grants:

Eighty-four percent of the F32 recipients have stayed in a science-related career, 68% have current job titles indicating active participation in research (e.g., PI, researcher, professor, instructor, or assistant professor), and 68% published research in a NIAMS mission-related field during the last 6 years. Thirty-four percent of F32 recipients have received R01 grants, and 59% had received some sort of NIH grant besides the F32. Eighty-seven percent of the recipients published during the last 10 years, and 59% published during the past two years. F32 recipients were primary authors in 33% of their 362 publications. Fifty-seven percent are active in at least one professional association, and 21% have received at least one professional award or honor.

There is limited diversity in the institutions receiving the F32 grants. During 1993-1994, the institutions receiving F32 grants were located in 20 states. However, four states received 50% of the grants ( California - 8, Massachusetts - 7, Texas - 5, Wisconsin - 3). The1993-1994 cohort of trainees were 56% male, 32% female, and 12% unknown gender. The recipients self-identified as 87% White, 3% Asian, and 1% Black or African American. None of the recipients self-identified as American Indian/Native American, Pacific Islander, or Hispanic. Nine percent were of unknown racial or ethnic origin. The average age of the trainees was 32 years old. Figures 12 and 13 show a summary of the outcomes for the F32 trainees.

Figure 12 F32 Publishing Outcome Summary

13 F32 Professional Outcome Summary

Figure 13 F32 Professional Outcome Summary

K01 Awards:

Overall, 100% of the K01 recipients have stayed in a science-related career, 100% have current job titles indicating active participation in research in a university setting (e.g., Assistant or Associate Professor), and 100% published research in a NIAMS mission-related field during the last six years. Five out of six of the K01 recipients have received R01 grants. One hundred percent of the recipients published during the last 10 years, and 83% published during the past two years. K01 recipients were primary authors in 38% of their 79 publications. Fifty percent are active in at least one professional association, and 17% have received at least one professional award or honor.

Of the six K01s awarded , three were in Tennessee , one in Illinois , one in New Hampshire , and one in Arizona . Because this cohort was so small, the demographic information on award recipients has been combined with the K08 data. Figures 14 and 15 below show the summarized outcomes for the K01 trainees.

Figure 14 K01 Publishing Outcome Summary

Figure 15 K01 Professional Outcome Summary

K08 Awards:

Overall, 100% of the K08 recipients have stayed in a science-related career, 62% have current job titles indicating active participation in research (e.g., PI, researcher, professor, instructor, or assistant professor), and 84% published research in a NIAMS mission-related field during the last six years. Fifty-five percent of the K08 recipients have received R01 grants, and 69% have received some sort of NIH grant besides the K08. Ninety-eight percent of the recipients published during the last 10 years, and 85% published during the past two years. K08 recipients were primary authors in 40% of their 613 publications. Ninety-one percent are active in at least one professional association, and 59% have received at least one professional award or honor.

There is limited diversity in the institutions with the K08 award recipients. During 1996-1997, the institutions receiving K08 grants were located in 20 states. However, five states received almost 50% of the grants ( Massachusetts - 11, New York - 5, California - 4, Pennsylvania - 4, Missouri - 4). The K08 and K01 recipients were 51% male, 45% female, and 4% unknown. They were self-identified as being 76% White, 10% Asian, 0% Black or African American, 2% American Indian/Native American, 2% Pacific Islander, 5% Hispanic (ethnic and racial), and 5% unknown. They had an average age of 37 years old. Figures 16 and 17 below show the summarized outcomes for the K08 trainees.

Figure 16 K08 Publishing Outcome Summary

Figure 17 K08 Professional Outcome Summary

The following sections of the report address each of the research questions in detail.

7.2 Has the training award recipient continued in a research career after training?

For each training award type, what proportion of trainees are conducting independent research, and what are their career choices, professional fields of study, and settings?

Based on the data collection methodology, this question can best be answered by comparing statistics on the percentage of trainees who have pursued scientific careers, their self-identified career job titles, the topics of the research publications authored by the trainees, and the settings in which the trainees are currently working. These data are shown in Tables 7-10 below.

A high percentage of trainees remained in science, ranging from 100% for the K awardees to 75% for the T32 recipients, as shown in Table 6. A career was classified as non-science if it consisted of: (1) going into business as an owner or manager; (2) a physician in private practice or working in a nonprofit hospital or clinical setting not affiliated with a university and having no recent supporting publications; or (3) other professions that appeared to have to relationship to research or an academic setting.

Table 7 Career Choices in Science

Grant Type	No. of Trainees	Science Career No. %	Non-Science Career No. %	Unknown Field No. %
T32	109	82 / 75%	5 / 5%	22 / 20%
F32	44	37 / 84%	3 / 7%	4 / 9%
K01	6	6 / 100%	---	---
K08	58	58 100%	---	---

The career choices by job title shown in Table 8 were often self-identified, and as such presented a challenge. For example, a professor may have been identified only by the professor title on his or her faculty website, but still be a Principal Investigator (PI) on a research project. Because of this, it is likely that the number of PIs is undercounted. However, to get a more accurate reading of the true number of trainees who have been PIs on a research project, a proxy was used for this category - the number of trainees who received R01 grants. Therefore, some of the trainees in the PI category are also found under other job titles in Table 7. In addition, a number of trainees had multiple occupations. For example, someone could be listed on their web site as both a Professor of Medicine and as the Director of the XYZ Institute. In these instances, the Professor occupation was used for purposes of tabulating the data. Similarly, if more than one job title was present (e.g., Assistant Professor of Medicine and Associate Professor of Surgery) the more senior title was used.

Table 8 Career Choices by Job Title

Grant Type	No. of Trainees	PI on an R01 Grant No. %	Researcher No. %	Prof. No. %	Instructor Asst. Prof.Assoc. Prof. No. %	Admin. or Mgt. No. %	Other (physician, retired, business) No. %
T32	109	19 / 17%	14 / 13%	7 / 6%	38 / 35%	4 / 4%	17 / 16%
F32	44	15 / 34%	9 / 20%	2 / 5%	19 / 43%	3 / 7%	3 / 7%
K01	6	5 / 83%	--	--	6 / 100%	--	--
K08	58	32 / 55%	3 / 5%	4 / 7%	28 / 48%	5 / 9%	17 / 29%

Note: Specific job titles could not be identified for 28 T32 and 6 F32 recipients, either because no information was available at all or there was no title listed with their place of employment.

Because much of the research being conducted by former trainees is basic scientific research, it was sometimes difficult to determine whether someone was conducting research in a NIAMS-related field from reading the titles and abstracts of their publications. If there were some doubt, but the paper was published in a journal with content in areas of interest to NIAMS, the research topic was considered NIAMS-related. Some of the research was cross-cutting (e.g., autoimmune reactions in children with diabetes) and was also assumed to be NIAMS-related if one of the topics was of interest to NIAMS mission areas. As shown in Table 9, some publications were not able to be identified specifically as either NIAMS or non-NIAMS related.

Table 9 Field of Study (proxy: topic of 3 most recent publications post-1999)

Grant Type	No. of Trainees	NIAMS Field		Non-NIAMS Field		Unknown Field
		No.	%	No.	%	No.	%
T32	109	60	55%	13	12%	11	10%
F32	44	30	68%	3	7%	5	11%
K01	6	6	100%	--	--	--	--
K08	58	49	84%	1	2%	3	5%

Note: 25 T32, 6 F32, and 5 K08 recipients were not identified as having published anything more recently than 1999.

The majority of trainees in all categories were found to be working in an academic setting. As shown in Table 10, of the trainees who were located, between 7%-14% were found to be working in the private sector. However, this could be understated, because it is likely that some of the trainees who were not located during the data search were not able to be found precisely because they are in the private sector. The assumption was made that if someone was working at a university-affiliated hospital and had published recent research papers, they were conducting research, even if they were not specifically listed as a faculty member of the university.

Table 10 Settings

Grant Type	No. of Trainees	Academic/ University		Government		Private		Non-Profit
		No.	%	No.	%	No.	%	No.	%
T32	109	61	56%	6	6%	15	14%	7	6%
F32	44	31	70%	3	7%	4	9%	2	4%
K01	6	6	100%	--	--	--	--	--	--
K08	58	48	83%	2	3%	4	7%	4	7%

Note: Current career settings for 20 T32 and 4 F32 trainees could not be found

Figure 18 below summarizes the responses to question 7.2. One anomaly in the data is the low percentage of K08 recipients who self-identified as researchers or professors. However, a high percentage of the K08 recipients (20%) were identified as being physicians. Since 83% of the K08 recipients are in an academic situation or affiliated with a university, it's possible that many are involved with clinical research in that setting but are not the Principal Investigators on research projects or are primarily teaching or practicing in hospitals affiliated with medical schools.

Figure 18 Research Career

7.2.1 For those trainees involved in research, is the trainee an individual whose career is primarily focused on research, including full-time researchers in academia, industry, and government as well as research administrators; or is the trainee an individual whose research is a secondary focus in their career, including educators in a research environment and clinicians that contribute to research by others?

It was not possible to answer this question from the collected data. One method might have involved determining whether the trainees were co-PIs on grants, but this information was not available in the IMPAC II database. The primary difficulty on this question was that most of the websites for the trainees did not give any indication of the level of involvement in research.

7.3 Has the trainee become a productive independent researcher?

Nine variables were used to attempt to answer this question. These included whether the trainees ever received any type of R grant from NIH, over what period of time an R01 award was received, the field of study for the grant, the proportion of trainees who participated in research grants but were not PIs, the proportion of trainees published in peer reviewed journals, the dates of publication, how frequently the trainees were the primary author, the proportion of trainees active in professional associations, and the proportion of trainees that received honors for their research. Of these, data for two of the variables could not be found. These were the field of study for the grants and the proportion of trainees who were participants in research but not PIs. Tables 10 - 16 contain the results of the data collection on the seven variables for which information was available.

7.3.1 For each training award type, what proportion of trainees has received independent research funding (e.g., private or public)?

It was difficult to find consistent information on private sources of funding, but it appeared that many of the researchers were getting grants from foundations and associations. Because consistency was problematic, Table 11 shows only R grants received from NIH, which were available from the IMPAC II database through the Training Assistant (TA) software. While the NIH database has an unknown number of data entry errors in it, these appear to be small in number and wouldn't significantly affect the data.

Table 11 Trainees Receiving R Grants

Grant Type	No. of Trainees	R01		Other R		Other NIH
		No.	%	No.	%	No.	%
T32	109	19	17%	7	6%	15	14%
F32	44	15	34%	7	16%	4	9%
K01	6	5	83%	--	--	--	--
K08	58	32	55%	4	7%	4	7%

7.3.2 For what period of time was funding received?

As shown in Figure 19 below, the number of R01 grants expiring in a particular year increases as the years progress. This indicates that trainees were getting their grants in larger number beginning in about 2000. A large proportion of the grants expire in 2007. However, at the time of this study, it was not possible to determine what percent of these grants were being renewed.

Figure 19 Year R01 Expires

7.3.3 What proportion of trainees has published in academic and peer-reviewed journals?

Table 12 shows the number of trainees who have published in academic and peer reviewed journals during the last 10, 5, and 2 years. These dates were picked because they show whether the trainees have remained in research. As the table shows, the amount of publishing has dropped over time. Some of the trainees published only once, during the time they were receiving the training grants. This table does not distinguish the number of times each trainee may have published or whether they were the primary author, which is contained in other tables that should be looked at in conjunction with Table 12.

Table 12 Trainees Published in the Last 10, 5, or 2 Years

Grant Type	No. of Trainees	Published in last 10 years		Published in last 5 years		Published in last 2 years
		No.	%	No.	%	No.	%
T32	109	85	78%	66	61%	54	50%
F32	44	40	87%	34	74%	27	59%
K01	6	6	100%	6	100%	5	83%
K08	58	57	98%	53	91%	49	85%

7.3.4 What are the dates of publication?

Table 13 shows the total number of papers or articles published that were authored or coauthored by training grant recipients in each year since the grant or award was received. The number of publications has risen, indicating increasing productivity for those remaining in research fields.

Table 13 Publications by Training Assistance and Year

Year	Grant Type
	T32	F32	K01	K08
2007	--	--	--	2
2006	111	62	10	148
2005	129	49	13	131
2004	84	53	16	85
2003	81	43	6	62
2002	53	27	6	39
2001	62	25	9	33
2000	59	24	6	28
1999	44	14	3	35
1998	49	14	6	31
1997	62	15	4	13
1996	65	16	--	2
1995	61	13	--	1
1994	46	4	--	1
1993	21	3	--	--
Other	24	--	--	2
Total Publications	951	362	79	613

7.3.5 Of those who have published, how often were they the primary author and how often a contributor?

Table 14 shows the total number of trainees who have published research articles or papers, the total number of publications, and of those publications, what proportion of the time the trainees were the primary author, and what proportion they were a contributor to the article but not the first or second author listed. This study did not identify instances where the trainee may have been the last author listed, but it was not expected that this would have occurred frequently this early in the research careers of the trainees.

Table 14 Authorship of Articles

Grant Type	No. of Trainees	No. Published Trainees	No. of Pubs	Primary Author	Contributor
		No. %	No.	No. %	No. %
T32	109	102 / 94%	951	427 / 45%	524 / 55%
F32	44	41 / 93%	362	118 / 33%	244 / 67%
K01	6	6 / 100%	79	30 / 38%	49 / 62%
K08	58	57 / 98%	613	243 / 40%	370 / 60%

7.3.6 What proportion of trainees is active in professional associations in their field?

While the professional association membership directories reliably indicated who members were, they did not reliably indicate whether the member had held an office or other leadership position within the association or how active the participation was. Therefore, Table 15 is an indication of membership only.

Table 15 Professional Association Memberships

Grant Type	No. of Recipients	Active in at Least One Professional Association
		No. %
T32	109	63 / 58%
F32	44	25 / 57%
K01	6	3 / 50%
K08	58	53 / 91%

Note: Many of the trainees are active in more than one association.

7.3.7 What proportion of recipients of NIH training support were also recipients of major honors for research achievement (e.g., from professional societies) in NIAMS area of responsibility?

Professional awards and honors may be overstated. Some awards are for designations such as " America 's Top Doctors" listing. Other honors appeared to be grants and fellowships received from foundations and associations. Some were difficult to distinguish as being related to research because they were titled, for example, "Teacher-Scholar of the Year" or "Outstanding Mentor" award. Table 16 shows the number and proportion of trainees who received at least one major award related to their professional achievement.

Table 16 Award Recipients

Grant Type	No. of Recipients	At Least One Award or Honor
		No. %
T32	109	23 / 20%
F32	44	9 / 21%
K01	6	1 / 17%
K08	58	34 / 59%

Figure 20 shows five of the nine variables investigated to determine whether the trainees had become productive, independent researchers.

Figure 20 Independent Researcher Variables

7.4 Is the trainee working in a field relevant to NIAMS mission?

7.4.1 Do NIAMS supported trainees continue to conduct NIAMS-related research throughout the course of their careers?

As included in Table 9, the data showed that 55% of the T32 recipients, 68% of the F32 recipients, 100% of the K01 recipients, and 84% of the K08 recipients continued to conduct research in NIAMS related areas, based on the subject of research results in the three most recent publications of trainees that were published after 1999.

7.5 Is there institutional diversity in the training grant program?

7.5.1 What is the geographic location of the institution receiving the award?

The states with recipients receiving between 1.9% and 0.5% of the grant pool were Georgia , Washington , Indiana , Maryland , Alabama , Iowa , New Hampshire , Vermont , Arizona , Florida , Mississippi , New Jersey , Oklahoma , Oregon , and Virginia . In all, 30 states had grant recipients. However, recipients in fifteen states received 84% of the grants and in five states received 50% of the grants as shown in Figure 21 below. Twenty states had no grant recipients.

Figure 21 Geographic Distribution of Grants by Percent

Figure 22 illustrates the above distribution percentages on a map of the United States to highlight the geographic gaps in the allocation of grants.

Figure 22 Mapped Geographic Distribution of Grants by Percent

7.5.2 What is the size of the institution receiving the award?

As shown in Table 17 below, the institutions consist mainly of universities and affiliated hospitals. They are primarily large, well known institutions.

Table 17 Institutions Receiving T32 Awards

Boston University Medical Campus	MA
Brigham and Women's Hospital	MA
Case Western Reserve University	OH
Duke University	NC
Emory University	GA
Harvard University Medical School	MA
Hospital for Special Surgery	NY
Loma Linda University	CA
Massachusetts General Hospital	MA
Medical College of Wisconsin	WI
New England Medical Center Hospitals	MA
New York University	NY
Rockefeller University	NY
Scripps Research Institute	CA
Stanford University	CA
Thomas Jefferson University	PA
U. of Texas Health Center , San Antonio	TX
University of Alabama Birmingham	AL
University of California at San Diego	CA
University of California , San Francisco	CA
University of Colorado Denver/HSC Aurora	CO
University of Connecticut	CT
University of Iowa	IA
University of Michigan , Ann Arbor	MI
University of Minnesota Twin Cities	MN
University of North Carolina , Chapel Hill	NC
University of Pennsylvania	PA
University of Rochester	NY
University of Texas Health Science Center at San Antonio	TX
University of Texas SW Med Ctr/Dallas	TX
University of Washington	WA
Vanderbilt University	TN
Virginia Commonwealth University	VA
Washington University	MO
Yale University	CT

7.5.2 Which T32 institutions also support trainees with F32 grants?

Table 18 below lists the institutions that receive T32 grants and also host trainees with F32 grants. There were 37 such institutions during 1993-1994, the years from which the sample cohorts were drawn. Of these, seven institutions were located in Massachusetts , six in California , four in Texas , two in North Carolina , and two in Tennessee . The remaining states each had one institution.

Table 18 Institutions Supporting F32 Awards

Baylor College of Medicine	TX
Beth Israel Deaconess Medical Center	MA
Brandeis University	MA
California Institute of Technology	CA
Cedars-Sinai	CA
Children's Hospital, Boston	MA
Colorado State University , Ft. Collins	CO
Columbia University , Morningside	NY
Dartmouth College	NH
Duke University	NC
Good Samaritan Hospital and Medical Center , Portland	OR
Harvard University Medical School	MA
Johns Hopkins University	MD
Massachusetts General Hospital	MA
Massachusetts Institute of Technology	MA
Mayo Clinic College of Medicine, Rochester	MN
Purdue University , West Lafayette	IN
Rutgers	NJ
Scripps Research Institute	CA
Stanford University	CA
Thomas Jefferson University	PA
University of California , Berkeley	CA
University of California , Davis	CA
University of Illinois at Chicago	IL
University of Massachusetts , Medical School Worcester	MA
University of Michigan	MI
University of North Carolina Chapel Hill	NC
University of Pennsylvania	PA
University of Tennessee Health Sciences Center .	TN
University of Texas Austin	TX
University of Texas Health Sciences Center , Houston	TX
University of Texas M.D. Anderson Cancer Center	TX
University of Vermont & St Agric College	VT
University of Washington	WA
University of Wisconsin - Madison	WI
Vanderbilt University	TN
Washington University	MO

7.5.3 What are the demographic features of the grant recipients?

The characteristics of the grant and award recipients are shown in Table 19 below. For the T32 recipients, these are the characteristics of the entire 1993 -1994 cohort, rather than just the subsample, which could not be separated out of the aggregate data. In addition, the characteristics of the K01 and K08 recipients were combined due to the small size of the K01 cohort. The number of trainees for each grant type is greater than the number shown in Table 3 because several of the grant recipients had gotten grants in both years of the cohort, and the aggregate data, which was provided by the NIH Division of Information Services from the IMPAC II PUB files, did not eliminate duplicate listings from multiple years.

Table 19 Demographic Characteristics of Trainees

Grant Type	Total	W	B	A	A I	P I	H	Unk. RNO	Avg. Age	M	F	Unk. Sex
T32	371	48%	2%	19%	0	0	3%	28%	34	58%	42%	0
F32	71	87%	1%	3%	0	0	0	9%	32	56%	32%	12%
K01/K08	107	76%	0	10%	2%	2%	5%	5%	37	50%	44%	6%

W = White B = Black/African American, A = Asian, AI = American Indian/Native American

PI = Pacific Islander, H = Hispanic (either as an ethnicity or race), RNO = Race or National Origin, M = Male, F = Female, UNK = Unknown

Table 20 shows the percentage of the T32 and F32 trainees that had M.D.s, Ph.D.s, or advanced other degrees.

Table 20 Advanced Degrees of T32 and F32 Trainees

Grant Type	M.D.	Ph.D.	M.D./Ph.D.	Other	Unknown
T32	50%	29%	6%	3%	12%
F32	26%	52%	2%	7%	13%

8.0 Qualitative Findings

8.1 Overview

The general research questions for the interviews were:

What are the strengths of the current program?
What are the weaknesses of the current program?
What are potential barriers to success?
If the current structure of the training program is deemed to be successful in developing and maintaining the research pipeline, are there modifications that can be made to enhance the effectiveness of the training programs offered at NIAMS?
If not, should one or more of the training programs be significantly modified to meet the needs and opportunities within the current research environment?

The interview probed these questions in more detail, and interview participants were asked to identify variables that could be used to define a successful career outcome for individual training grant and career award recipients.

Data were also collected on the individual backgrounds of the interview participants regarding their experiences at other Institutes and Centers at the NIH or as grant recipients during other stages of their research careers. Several of the participants had received either a T32 or F32 grant. Overall, their perspective was that the NIH was not identified as a funding source for the T32, nor did the trainee typically make the connection to NIH. The descriptions of program quality varied widely, ranging from structured programs where the institution supporting the T32 trainees had activities, such as seminars, to collectively train and mentor the T32 recipients versus programs where there was no distinction between a T32 recipient and someone receiving other forms of financial aid from the institution, as well as no special training activities. However, the participants who had received T32 grants thought that the financial support was critical in helping them to continue their research related career.

The F32 recipients had a very positive view of their F32 experience, although they reported varying quality of the mentoring they received. However, all F32 recipients thought the funding came at a critical point in their careers and allowed them to continue in a research related field.

Below are high level summaries of the common themes that were found in the interview participant responses. They are organized around each grant and career development award type and the overall program.

8.2 T32 Grants

The identified strengths of the T32 grants were that they: (1) create an overall training environment that is supportive to trainees; (2) e xpose people early to areas of interest to NIAMS; and (3) are f lexible for participating institutions in regard to program management, selection of trainees, and allocations to PIs. However, the program was identified as having some weaknesses. These were: (1) v ariability in the quality of programs between institutions is hard to track due to the manner and type of information that is reported back to NIAMS; (2) many of the trainees do not know that their T32 funding actually comes from NIAMS, they think it is from the school; and (3) there is a need to increase the scope of the grants to include more emphasis on interdisciplinary, translational, and clinical research, and this should be encouraged in the application review process.

When asked to define success for T32 trainees, there were a range of responses. It was generally acknowledged that there would be degrees of success. Clearly not everyone would get an R01 grant. Generally, the participants agreed that trainees at a minimum would get exposure to research, regardless of their later career choice. This in itself would be an important accomplishment of the program, even if the trainee later became a science writer, teacher, or program administrator, rather than a researcher. Another suggestion was to look at the NRSA criteria themselves for guidance on what the program considers an acceptable outcome. The NRSA program specifies criteria for payback in the event the trainee does not successfully complete the program. Under these guidelines, acceptable careers include research, teaching, and research administration.

The primary barrier to success for T32 trainees was the same barrier faced by all grant and award recipients. That is, the research pipeline is actually a funnel. There are not enough tenured positions or R01 grants for all recipients who are interested in research careers. This is especially acute for T32 recipients, because they are the largest group of recipients, and they are at the beginning of their careers. In addition, interview participants pointed out that the citizenship/permanent resident requirements limit the applicant pool. Many promising researchers from foreign countries do not qualify for T32 support. Rather, they are supported by other mechanisms, such as becoming research assistants on R01 grants. There was some questioning as to whether this situation results in the T32s not being awarded to the most promising trainees, but rather to students who meet the citizenship requirements.

Another area explored by the participants was the trainee-mentor relationship. NIAMS participants that work most closely with the T32 grants indicated that this relationship may not be fully utilized to the trainee's benefit. There is variable quality to the training programs, because they are run differently at every institution. Some institutions run extensive group activities for T32 (and other) trainees and have successful mentoring programs. Other institutions tend to treat the T32 recipients as extra labor in the labs. While the latter approach provides on-the-job training, it does not take full advantage of all the T32 has to offer. Participants reported varying experiences with T32 grants if they had themselves been recipients earlier in their careers, confirming that the programs can be quite different from institution to institution.

Participants whose work involves a lot of contact with T32 institutions suggested modifications to improve the T32 program. These included: (1) adopt a more broadly based approach in the Scientific Review Section to emphasize professional development and institutional diversity criteria; (2) link up the trainees with other programs such as the Clinical and Translational Science Award (CTSA) program, Clinical Research Centers (CRC), or other Centers to broaden their experiences; (3) financially support universities through the T32 grants that want to develop on-line training courses that could be shared between institutions to give a more consistent experience to trainees; (4) give the trainees exposure to NIH and NIAMS to motivate them to continue in research by bringing them on-site for special programs; and (5) focus on good mentors: look for them, support them, train them, and encourage the institutions to put more effort into developing the mentor-trainee relationship.

8.3 F32 Grants

Overall, there was unanimity among the participants that the F32 program is highly valuable. This was primarily because it is competitive, which makes it prestigious. It provides good practice to individuals for applying for grants and is a first step toward being an independent researcher. Other strengths of the program mentioned by NIAMS participants were that: (1) the outside independent review of applications at CSR is unbiased; and (2) the generous award size is sufficient to attract applicants at that stage in their careers. However, it was pointed out that particularly for M.D.s and those with student loan debt, the stipend may be too low to attract them. (Note: 56% of the T32 recipients and 28% of the F32 recipients in this study were M.D.s.) Additional weaknesses of the program were: (1) the mentor-trainee relationships have variable quality; and (2) the funding should last longer than three years, because people need more time to prepare for independence.

Because the F32 is more competitive than the T32, the participants' definition of success for trainees was a little narrower. Simply being exposed to research, while important, was not deemed sufficient by most of the participants. Rather, the F32 trainees were expected to get or attempt to get an R01 grant and go into research or teaching as a career choice. However, a small number of participants expressed the view that simply having the exposure to research was important and could be considered a successful outcome, regardless of future career choice.

The identified barriers to success for F32 trainees were very similar to the T32 barriers. Lack of future funding and tenured positions, citizenship requirements, and the variable quality of the trainee-mentor relationship were all cited by participants. There was also some thought that the low stipends keep out M.D.s, and that the trainee's lack of knowledge about other funding opportunities may hinder their future success.

Suggested modifications to the F32 grant program included: (1) expanding the size of the program to award more grants; (2) adding a fourth year for those who need more time to successfully apply for more independent funding; (3) relaxing the citizenship/permanent resident requirements in order to open the pool to the most promising researchers; (4) increasing NIAMS' focus on developing good mentors for the trainees; (5) bringing the trainees to special programs at NIH or NIAMS to motivate them to stay in a research career; and (6) creating a bundling effect where multiple F32s are located at institutions and centers in order to take advantage of group support and activities, similar to what is experienced at the best T32 institutions.

8.4 K01 Awards

Participants expressed the most mixed views about the K01 award. Some of the participants did not feel it was necessary in light of other K awards, while others were supportive. The strengths of the K01 program were identified as its ability to: (1) support development of multidisciplinary skills and broader experience for the trainee; (2) create partnerships with universities to support recipients for five years; and (3) give people time to achieve tenure. In fact, supporting trainees while they attempted to get tenure and secure an R01 grant, a process that could take several years, seemed to be the number one reason people supported the awards. On the other hand, this strength was also cited by a few participants as a weakness. That is, the award supports people who may not be the strongest researchers, as evidenced by the amount of time it takes them to become independent. Other weaknesses of the K01 award were that (1) the Scientific Review Section process was seen as being too subjective; (2) the citizenship/permanent resident requirements limit the applicant pool; and (3) the award has vague eligibility criteria and can be confused with other K awards.

The definition of success for K01 trainees was much narrower relative to the T32 and F32 grants. The K01 recipient was clearly expected to get or attempt to get an R01, go into research or teaching, and make a contribution to science. Barriers to accomplishing this were the tight competition for research funds and tenured faculty positions and the low salary support. In addition, the limited number of mentors in translational and clinical research was also cited as a problem. The participants did not believe that a trainee would succeed if he or she did not have a good mentor at this stage in their career development.

Some suggested modifications to the K01 awards were to: (1) clarify the amount of time per week that must be devoted to research (75% of 40 hour week) so that trainees could accomplish other work with other funding sources; (2) raise the stipend level or work with foundations to help supplement funding for the trainees; (3) work with institutions to develop more bridge funding until tenure is achieved; (4) eliminate the citizenship requirements; and (5) focus on finding good mentors and providing mentor training.

8.5 K08 Awards

The K08 awards were viewed by the participants as being a critical part of the research pipeline. The strengths of the K08 awards were that they were considered to be prestigious and critical for the development of physician scientists. The five years of support from the award were viewed as creating commitment to the trainee from universities and helping them to build partnerships. For physicians, the award provides the opportunity to conduct research rather than see patients all the time.

The weaknesses of the K08 awards were primarily that the time commitment to research combined with the low stipend discourages physicians from applying, because there are too many restrictions for the program to be attractive to M.D.s.

The definition of success for K08 trainees included getting or attempting to get an R01 grant, establishing a research career, and making a contribution to science. A minority of the participants thought that as long as the K08 trainee supported research in some way, whatever the career choice turned out to be, the program would have been a success. In addition, another indicator of success, having a good match between the number of applicants and the need for researchers was mentioned.

The barriers to success for K08 trainees were the same as for the other grants and awards regarding the scarcity of research funds and tenured positions, as well as the citizenship/permanent resident requirements. In addition, the lack of university support for time devoted to research rather than clinical practice was frequently cited. This was related to the low salary support provided to the trainees. Finally, as is the case with the K01 awards, the limited number of mentors in translational and clinical research was viewed as problematic for the ultimate success of the trainees.

Suggested modifications to the K08 awards included: (1) clarifying the amount of time per week that must be devoted to research (75% of a 40 hour week); (2) raising the stipend level; (3) looking for multidisciplinary involvement on the part of the applicants; (4) looking at the review process to make sure clinicians aren't ranked like bench scientists; (5) eliminating the citizenship requirements; and (6) focusing on good mentors and mentor training.

8.6 Overall Program

In looking at the overall program, the interview participants identified the strengths as being: (1) it supports the next generation of researchers; (2) there is good balance between programs; and (3) because there are several mechanisms available, the program is able to reach a variety of people and fill various needs along the career path.

The interview participants also identified the following weaknesses of the overall program: (1)

the programs need to focus on management training as well as the actual experiments - project management, budgeting, staffing are important skills for PIs; (2) there should be greater emphasis on translational research in the application review process; (3) the program needs greater emphasis on mentoring; and (4) there is a need for more comparability in the application review process between study sections. In addition, the participants observed that many universities' expectations for tenure are not realistic - not everyone will get an R01 grant.

Participants suggested modifications to the overall program, which included (1) putting more emphasis on interdisciplinary approaches when awarding grants; (2) linking training programs with research centers to get more synergy and cross-training; (3) conducting a separate examination of the NIAMS application review process; (4) providing help to institutions in order to harmonize programs so that all grants are maximized; e.g., if an institution supports recipients of several types of grants, there should be interaction between the programs; (5) focusing on good mentors and mentor training as part of grant oversight; and (6) looking to fill gaps in the research pipeline when awarding grants.

9.0 Additional Inputs to the Evaluation

9.1 NIAMS Training Grants Review

Dr. Helen Lin of the NIAMS Grant Review Branch provided information on the grant review process for training grants. The NIAMS Arthritis and Musculoskeletal and Skin Diseases (AMS) Study Section reviews T32 training grants, and K01 and K08 career development awards. The F32 training grants are reviewed outside of NIAMS at the NIH Center for Scientific Review. The AMS Study Section is composed of rheumatologists, dermatologists, bone biologists, muscle biologists, and orthopedic surgeons. It is currently chaired by one of the members of the working group to this evaluation.

The review criteria for the mentored K applications consider the candidate, their career development plan, their research plan, their mentor, and the institutional environment and commitment. The qualifications of the candidate and mentor, along with the career development plan and institutional environment and commitment are collectively more important than the research plan. The review criteria for T32 applications consists of the program design, the program director, the training faculty, the trainees, the research environment, and the training record of the institution.

As shown in Figure 24, generally over one-third of the K01 applicants have over eight years of postdoctoral training at the time they submit their applications. About 70% of the K01 applicants have five or more years of postdoctoral training. Figure 25 shows that between 84-95% of the K08 applicants had at least 5-6 years of postdoctoral training before submitting their applications. During 2004 and 2005, over half the K08 applicants had seven or more years of training.

Figure 23 Years of Postdoctoral Training Before K01 Submission

Helen Lin

Figure 24 Years of Postdoctoral Training Before K08 Submission

Helen Lin

Dr. Lin also provided information showing that of a sample of 25 K01 awardees, 12 received R01 grants, taking up to six years to receive the first grant. Of the 13 that never received an R01 award, five were still applying at the end of the same six year period. She indicated that K awardees have a 38% success rate for R01 grants as compared to the 20% success rate overall for R01 grants.

Additional information supplied by Dr. Lin indicated that in FY 2000, there were 44 K08 awardees funded by NIAMS. Of these, 33 applied for R01 grants and 22 of these were successful. Among the 22 R01 awardees, nine got the R01 grant on the first attempt, six got the grant on the second submission, and seven were successful on the third submission.

9.2 Current Training Grant and Career Award Activities at NIAMS

Dr. Madeline Turkeltaub, NIAMS EP Deputy Director, and Dr. William Sharrock, a NIAMS EP program director, provided data on the current status of NIAMS Training Grants and Career Award activities at NIH. Preliminary analysis of data indicates that researchers progress to individual awards more frequently if they receive an institutional award. Also, because K01 recipients have a 50% success rate in R grant applications, this reinforces the perceived value of having a K grant available between T32 and R grants. In addition, the peer review boards already factor in years of postdoctoral experience and the reasons behind lengthy tenure of applicants into the existing review process, so that a cap on the number of years of postdoctoral experience is not necessary.

NIAMS is pursuing several courses of action to increase the diversity of the applicant pool, including:

Accepted applications for the K99 award for the first time and will award four of them this year.
Increased support to M.D. K99 recipients up to $75,000 per year for the first two years.
Developed public/private partnerships with the American Skin Association for F32s with the goal of increasing the pool of applications and subsidizing funding for successful applications.
Changed time requirement for K02s and K08s to 50% effort devoted to research, down from 75%.
Considering pulling awards together to provide better mentoring experiences and grouping of educational courses as CTSAs have become more active, and include a training component.
Targeting a new NIH Director's Award - the New Innovator Award Program - toward investigators who do not have the preliminary data for the R01 application process. NIH will give 14 of these awards. Evaluation criteria are not yet established.
Review by NIH of pay mechanisms and investment in K awards.

9.3 American College of Rheumatology (ACR) Comments

Dr. David Wofsy, the chairman of the working group provided information collected by the ACR involving academic rheumatologists. Key findings included:

The percentage of rheumatologists receiving independent grant support has declined from 41% to 22% since 1997.
A significant amount of time - most commonly seven to eight years - is needed for academic rheumatologists to achieve independent investigator status. Sixty-five percent take in excess of nine years.
Insecurity related to grant funding is a major reason why rheumatologists leave academic life and pursue clinical practice.

Based on these findings, the ACR advocated two broad recommendations for grant funding going forward:

Maintain a diverse portfolio (including training grants, career development grants, and early independent grants).
Establish coordinated strategy and complementary programs with other funding agencies such as NIH and research foundations.

These findings and recommendations reinforce a perception that exists among many researchers and trainees that rheumatology research is not a viable career pathway because of the lack of grant support. Training programs are valuable in recruitment, but do not currently address this issue sufficiently to eliminate this perception of risk.

9.4 Qualitative Peer Feedback

Several working group members provided input based on their informal polling of colleagues. Nearly all saw the value added by the T32 grant program, particularly as an integrated program that exposed researchers to experiences and interaction outside the lab setting. Additionally, several members reported that the T32 grant was a powerful recruiting tool and differentiator between academic research programs. The working group consensus was that the T32 is an essential and irreplaceable aspect of many programs that provides a mechanism to support researchers in their second year. This critical role of the T32 in getting researchers - particularly those without data or a Ph.D. - into a lab setting is becoming more prominent as other funding sources are diminishing.

Discussion of other peer feedback touched on a variety of topics, including:

The value of evaluating individual mentors in the K award process and the possibility of integrating this metric into the T32 evaluation process.
The need to supplement K awards with other sources of funding to relieve the pressures created by the amount of time that needs to be devoted to research.
The need to analyze the impact of T32s on the ability to successfully obtain R01 grants, particularly in evaluating those in a clinical training program versus their non-T32 peers.
The value of academic mentorship and the need to foster good mentors with training or other support.
Strategies for integrating clinician scientists into a training environment by getting more clinicians to apply for training grants. The value of including the whole team - clinicians as well as laboratory scientists - in the training environment and encouraging interaction.

10.0 Working Group Findings

10.1 Overall Success of the Program

The working group concurred with the approach of evaluating the different NIAMS programs independently, as these programs each have different objectives and target audiences. The data indicate that the four NIAMS programs are successful on the whole. Although the training sequence funded by NIH is often referred to as a pipeline, it is, in fact, more of a funnel. There are more trainees at the early stages of their research careers, such as in the T32 program, than there are in the later mentoring stages found in the K career award program. But even getting a K award is no guarantee of being able to secure an R01 grant. Therefore, it is to be expected that lower percentages of T32 recipients would eventually win an R01 grant than K award recipients. Of particular interest were the data on publishing, since this is a good indicator of involvement in research regardless of the funding source.

The working group's definition of success, then, for a T32 recipient was the broadest. This was generally in conformance with the views of most of the NIAMS EP program directors and grant management and review staff that were interviewed. Because it is not possible for every T32 recipient to become an independent researcher with an R01 grant, the benefit of the program is that T32 recipients are exposed to scientific research methods and may apply the knowledge and experience gained in any number of related pursuits. Nevertheless, 17% of the T32 recipients did receive R01 grants, and 37% received some sort of NIH research grant after their T32 award. The percentage of T32 recipients that are still working in scientific fields was quite high (75%), and 55% currently have jobs that indicate active participation in research. Also, 50% of the trainees had published during the past five years. Overall, the working group indicated that this was a good outcome for the cohort of T32 grantees.

Because the F32 grants are awarded to individuals, the working group had higher expectations of success for these trainees than for the T32 recipients. Overall, 34% of the F32 recipients went on to receive R01 grants and 59% had received some sort of NIH research grant after their F32 award. In fact, 84% of the trainees stayed in a science-related career, 73% have current job titles indicating active participation in research, and 68% had published research in a NIAMS-related field during the last six years. When combined with the other data on F32 trainees, the working group indicated that the overall success of the F32 program was satisfactory.

The working group had the highest expectations for the K award cohorts. In fact, five out of six of the K01 recipients have received R01 grants, and 55% of the K08 recipients have received R01 grants. The working group speculated that the success rate for K08 awards (100% retention in science-related careers) may actually be an indicator that K awards are not available in sufficient numbers and that the need may substantially exceed the program size. Paradoxically, a somewhat lower success rate might provide a stronger indication that the program has reached the people it should. In addition, given that K08 applicants are typically at a stage of their careers where alternative sources of support cannot be sustained through a series of NIH application cycles, the lengthy cycle for resubmitting K awards is an area needing improvement. Addressing K award issues should be a priority due to the data illustrating that the K awards are successful in creating new investigators.

10.2 Future Studies

The working group noted that future evaluation efforts that are prospective, rather than retrospective, should include a control group in the design. Possible control groups to consider include:

Clinical trainees with T32s compared to others from the same clinical training programs without T32s
Trainees receiving support from other NIH Institutes compared to NIAMS trainees
Training grant recipients compared to training grant applicants who did not receive training grants

The working group also suggested additional data to consider collecting and analyzing in future studies including:

More in-depth and clearly defined data on qualitative measures of success: What is the quality of the trainee's contribution to science? What is the relative value of the content of publications, as well as the number of publications?
Comparative data on NIAMS trainee performance relative to other NIH Institutes
Retrospective look at new R01 awardees who would have been eligible for Ks: Did they have Ks, Fs, and Ts in their training?
Better data on long-term retention by examining cohorts that precede the 1980s
Possible common trends from examination of "unsuccessful" recipients to identify predictors of not staying in research
Comparison of career outcomes with unsuccessful applicants
More direct input from key NIAMS personnel with intimate knowledge of the programs and recipients - existing qualitative input somewhat homogenized
Potential subsets within each cohort that are most successful, such as arthritis researchers or musculoskeletal researchers

11.0 Working Group Recommendations

R1: Establish a structured data collection mechanism to support ongoing evaluation of training grant effectiveness by type of grant.

R2: Acknowledge the economic aspects of research by providing more flexibility on the percent effort required for K awards to accommodate clinical responsibilities and other personal and professional circumstances and by lifting restrictions that limit other sources of funding.

R3: Avoid imposing a time limit from completion of degree on applications. Maintain flexibility and discretion of the peer review board to reward outstanding candidates.

R4: Build on current success - illustrated by initial data collection - of the training grant and career development award program to leverage the recent increase of participants in NIAMS mission-related programs by increasing funding for NIAMS training grant mechanisms. The pipeline of researchers cannot be expanded unless the number of awards and the amount of funding is also increased.

R5: Consider integrating a new component into NIAMS institutional training grant strategy that would address the related dilemmas of prolonged training followed by multiple application cycles in pursuit of a K award, which were seen as major deterrents to a career in science.

Some committee members favored implementation of a new mechanism in which selected trainees with particularly high potential could be identified early in their training by the institution rather than by NIAMS and provided with K level support (in dollars and duration).
Some members of the committee favored implementation of a bridge-type award that would support trainees at a K level for 2-3 years after fellowship training while they sought a K award. One way of implementing this idea might be by the addition of junior faculty positions to established T32 programs so that selected trainees could be provided with financial support after completing their fellowship training. The support should be equivalent to a K award but perhaps of shorter duration, with the goal to provide a bridge while the individuals sought a K award.
Some members of the committee favored using whatever funds might be devoted to the two proposed mechanisms above to expand funding for the existing K programs and thus reduce the likelihood that an applicant might have to survive several application cycles before succeeding.

R6: Increase NIAMS budget for R01 grants so that there are more opportunities for trainees to conduct independent research at the end of the pipeline.

R7: Centralize training information to make information on different mechanisms more accessible to potential applicants. Encourage collaborative interaction with professional and constituent organizations to develop a robust complementary portfolio of training funding.

R8: Structure the criteria for success in grant review to encourage and reward integrated and interdepartmental approaches, foster innovation, and support interdisciplinary mentorship in applications. Reinforce the value of grant writing and management in program curriculum.

R9: Reinforce the value of mentorship by providing a range of opportunities (e.g., annual meetings at NIAMS, web-based modules, etc.) that support training of mentors as well as trainees and that foster an environment of collaboration and support for mentors and those being mentored.

R10: Work with other NIH Institutes and private foundations to insure that there is a comprehensive and complementary portfolio of funding mechanisms for trainees.

Appendix 1 Working Group Members

David Wofsy, M.D., Chair

Professor
Department of Medicine
School of Medicine
University of California , San Francisco
VA Medical Center (IIIR)
4150 Clement St.
San Francisco , CA 94121
Tel: 415-750-2104
Fax: 415-750-6920
Email: wofsyd@medsch.ucsf.edu

Alan Moshell, M.D.

Department of Dermatology
Washington Hospital Center
1955 Limb Tree Ln.
Vienna, VA 22182
Tel : 202-877-6654
Fax : 202-877-3288
E mail: moshella@cox.net

Paul D. Allen, M.D., Ph.D.

Professor of Anesthesia
Department of Anesthesia
Brigham and Women's Hospital
75 Francis St .
Boston, MA 02115
Tel: 617-732-7334
Fax: 617-732-6927
Email: allen@zeus.bwh.harvard.edu

Alice Pentland, M.D.

Professor and Chair
Department of Dermatology
University of Rochester
601 Elmwood Ave.
Rochester , NY 14642
Tel: 585-275-3871
Fax: 585-273-1346
Email: alice_pentland@urmc.rochester.edu

Anne Davidson, M.D.

Feinstein Medical Research Institute
350 Community Drive
Manhasset NY 11030
Tel: 516-562-2915
Email: adavidson1@nshs.edu

Sean P. Scully, M.D., Ph.D.

Cedars Medical Center
1400 NW 12 AVE
Suite 4035
Miami , FL 33136
Tel: 305.325.4683
Fax: 305.325.4784
Email: sscully@med.miami.edu

Milton Hernandez, Ph.D.

Director
Office of Special Populations and Research Training
National Institute of Allergy and Infectious Diseases
National Institutes of Health
6700B-Rockledge Dr.
MSC 7610 Room 2101
Bethesda , MD 20892-7610
Tel: 301-496-8697
Fax: 301-496-8729
Email: mhernandez@niaid.nih.gov

Elizabeth Shane, M.D.

Professor of Clinical Medicine
Columbia University
College of Physicians and Surgeons Building
630 West 168 th St .
PH8 West, Room 864
New York , NY 10032
Tel: 212-305-6528
Fax: 212-305-6486
Email: es54@columbia.edu

Appendix 2 Participating NIAMS Staff

Carl C. Baker, M.D., Ph.D.

Director, Skin Diseases Program
Division of Skin and Rheumatic Diseases
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
6701 Democracy Boulevard
Suite 800 , MSC 4872
Bethesda , MD 20892
Tel: 301-594-5017
Fax: 301-480-4543
Email: bakerc@mail.nih.gov

Cheryl Lapham, Ph.D.

Health Scientist Administrator
Division of Skin and Rheumatic Diseases
National Institute of Arthritis and Musculoskeletal
and Skin Diseases
National Institutes of Health
One Democracy Plaza
6701 Democracy Boulevard
Room 800, MSC 4872
Bethesda , MD 20892
Tel: 301-496-0810
Fax: 301-480-4543
Email: clapham@mail.nih.gov

Jonelle K. Drugan, Ph.D.

Science Policy Analyst
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
9000 Rockville Pike
31 Center Drive , room 4C13
Bethesda , MD 20892
Tel: 301-496-8271
Fax: 301-480-0183
Email: druganj@mail.nih.gov

Helen Lin, Ph.D.

Scientific Research Administrator
Scientific Review Branch
National Institute of Arthritis and Musculoskeletal
and Skin Diseases
National Institutes of Health
6701 Democracy Boulevard Room 800, MSC
4872
Bethesda , MD 20892
Tel: 301-594-4956
Fax: 301-402-2406
Email: linh1@mail.nih.gov

J. Elizabeth Gretz, Ph.D.

Director, Immunology and Inflammation Program Division of Skin and Rheumatic Diseases
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
6701 Democracy Boulevard Suite 800, MSC 4872
Bethesda , MD 20872
Tel: 301-594-5032
Fax: 301-480-4543
Email: gretze@mail.nih.gov

Anita Linde

Director
Office of Science Policy and Planning
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
9000 Rockville Pike
31 Center Drive , room 4C13
Bethesda , MD 20892
Tel: 301-496-8271
Fax: 301-480-0183
Email: lindea@mail.nih.gov

Stephen I. Katz, M.D., Ph.D.

Director
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
9000 Rockville Pike
31 Center Drive , room 4C32
Bethesda , MD 20892
Tel: 301-496-4353
Fax: 301-480-6069
Email: katzs@mail.nih.gov

Melinda B. Nelson

Chief, Grants Management Branch
Extramural Program
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
One Democracy Plaza
6701 Democracy Boulevard
Room 838, MSC 4872
Bethesda , MD 20817
Tel: 301-594-3535
Fax 301-480-5450
Email: nelson1@mail.nih.gov

Marie Mancini, Ph.D.

Health Scientist Administrator
Division of Skin and Rheumatic Diseases
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
One Democracy Plaza
6701 Democracy Boulevard
Room 882, MSC 4872
Bethesda , MD 20892
Tel: 301-594-5014
Fax: 301-480-4543
Email: mancinim2@mail.nih.gov

Glen H. Nuckolls, Ph.D.

Director, Muscle Disorders and Therapies Program
Division of Musculoskeletal Diseases
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
6701 Democracy Boulevard Suite 800 , MSC 4872
Bethesda , MD 20892
Tel: 301-594-4974
Fax: 301-480-4543
Email: nuckollg@mail.nih.gov

Joan McGowan, Ph.D.

Chief, Division of Musculoskeletal Diseases
Extramural Program
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
One Democracy Plaza
6701 Democracy Boulevard
Room 864, MSC 4872
Bethesda , MD 20892
Tel: 301-594-5057
Fax: 301-480-4543
Email: mcgowanj@ep.niams.nih.gov

James S. Panagis, M.D., M.P.H.

Director, Orthopaedics Program
Division of Musculoskeletal Diseases
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
6701 Democracy Boulevard Suite 800 , MSC 4872
Bethesda , MD 20892
Tel: 301-594-5055
Fax: 301-480-4543
Email: panagisj@ep.niams.nih.gov

Wilma Peterman Cross, M.S.

Deputy Director
Office of Science Policy and Planning
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
9000 Rockville Pike
31 Center Drive , room 4C13
Bethesda , MD 20892
Tel: 301-496-0803
Fax: 301-480-7934
Email: petermaw@mail.nih.gov

William Sharrock, Ph.D.

Director, Bone Biology Program
Division of Musculoskeletal Diseases
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
One Democracy Plaza
6701 Democracy Boulevard
Room 800 MSC 4872
Bethesda , MD 20892
Tel: 301-594-5103
Fax: 301-480-4543
Email: sharrockw@mail.nih.gov

Louise M. Rosenbaum, Ph.D.

Science Policy Analyst
National Institute of Arthritis and Musculoskeletal
and Skin Diseases
National Institutes of Health
9000 Rockville Pike
31 Center Drive , room 4C13
Bethesda , MD 20892
Tel: 301-496-8271
Fax: 301-480-0183
Email: rosenbauml@mail.nih.gov

Madeline Turkeltaub, RN, Ph.D., CRNP, FAAN

Deputy Director
Extramural Program
National Institute of Arthritis and Musculoskeletal
and Skin Diseases
National Institutes of Health
One Democracy Plaza
6701 Democracy Boulevard
Suite 800 MSC 6500
Bethesda , MD 20892
Tel: 301-451-5888
Fax: 301-480-4543
Email: turkeltm@mail.nih.gov

Susana Serrate-Sztein, M.D.

Chief, Division of Skin and Rheumatic Diseases
Extramural Program
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
One Democracy Plaza
6701 Democracy Boulevard
Room 876, MSC 4872
Bethesda , MD 20892
Tel: 301-594-5032
Fax: 301-480-4543
Email: szteins@mail.nih.gov

Yan Wang, M.D., Ph.D.

Director, Scientific Review Branch
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institutes of Health
6701 Democracy Boulevard Suite 800 , MSC 4872 Bethesda , MD 20872
Tel: 301-594-4952
Fax: 301-402-2406
Email: yw72q@nih.gov

Appendix 3 Additional Graphs and Tables With Results From Data Collection

Range and Frequency of Publications:

Current Career Settings:

TABLE OF CONTENTS

Listing of Figures

Listing of Tables

1.1 Overview

1.2 Findings

1.2.1 Definition of success

Individual Trainee Success

Training Program Success

NIAMS Success

1.2.2 Evaluation Findings

1 Summary of Key Findings by Grant Type

T32 Grants:

Figure 1 T32 Publishing Outcomes

Figure 2 T32 Professional Outcomes

F32 Grants:

Figure 3 F32 Publishing Outcomes

Figure 4 F32 Professional Outcomes

K01 Awards:

Figure 5 K01 Publishing Outcomes

Figure 6 K01 Professional Outcomes

K08 Awards:

Figure 7 K08 Publishing Outcomes

Figure 8 K08 Professional Outcomes

Qualitative Findings

1.3 Limitations

1.4 Working Group Recommendations

R1: Establish a structured data collection mechanism to support ongoing evaluation of training grant effectiveness by type of grant.

R2: Acknowledge the economic aspects of research by providing more flexibility on the percent effort required for K awards to accommodate clinical responsibilities and other personal and professional circumstances and by lifting restrictions that limit other sources of funding.

R3: Avoid imposing a time limit from completion of degree on applications. Maintain flexibility and discretion of the peer review board to reward outstanding candidates.

R5: Consider integrating a new component into NIAMS institutional training grant strategy that would address the related dilemmas of prolonged training followed by multiple application cycles in pursuit of a K award, which were seen as major deterrents to a career in science.

R6: Increase NIAMS budget for R01 grants so that there are more opportunities for trainees to conduct independent research at the end of the pipeline.

R7: Centralize training information to make information on different mechanisms more accessible to potential applicants. Encourage collaborative interaction with professional and constituent organizations to develop a robust complementary portfolio of training funding.

R8:Structure the criteria for success in grant review to encourage and reward integrated and interdepartmental approaches, foster innovation, and support interdisciplinary mentorship in applications. Reinforce the value of grant writing and management in program curriculum.

R9: Reinforce the value of mentorship by providing a range of opportunities (e.g., annual meetings at NIAMS, web-based modules, etc.) that support training of mentors as well as trainees and that foster an environment of collaboration and support for mentors and those being mentored.

R10: Work with other NIH Institutes and private foundations to insure that there is a comprehensive and complementary portfolio of funding mechanisms for trainees.

2.0 Background

2.1 Overview of Program

Table 2 Training Grant Funding by Fiscal Year ($ in m)

Table 3 Summary of Characteristic of Grants and Awards Studied

2.2 Need for an Outcome Evaluation

2.3 Outcome Evaluation Approach

2.4 Role of the Working Group of Outside Experts

2.5 Definition of Success

3.0 Conceptual Framework

Figure 9 Conceptual Framework

4.0 Study Research Questions

4.1 Quantitative

2a. Independent research funding

2b. Research publications

2c. Professional activities and recognition

4.2 Qualitative

5.0 Study Methodology

5.1 Quantitative

Approach

Table 4 Grant Cohorts

Table 5 Total Grants and Awards at NIAMS

Key Variables

Program resources

Population characteristics

Program activities

Program goals and performance measures

Other variables of interest

Areas in which there was interest in but the data were not consistent or were difficult to collect on individual trainees included:

Data Sources

Sampling

Table 6 Grant Recipient Cohorts Sample Size

Data Analysis

5.2 Qualitative

Qualitative Variables

Program activities

External factors

Data Sources

Data Collection Instrument

Data Analysis

Data Preparation

6.0 Limitations

7.0 Quantitative Findings

7.1 Overview

T32 Grants:

Figure 10 T32 Publishing Outcome Summary