User Guide and Scoring

Side view of a young businessman giving a presentation and pointing at the whiteboard

Organization of the TIP Tool

The TIP Tool 2.0 includes two main sections:

Section one gathers information about the training and its characteristics. This section focuses on descriptive characteristics, such as the target psychotherapies, target diagnoses, training format, and timing of the training. This part of the tool is not scored, as these features were not consistently associated with better outcomes in the literature. Instead, it is designed to provide context and background on the training.

Section two comprises the scored domains of the tool. The section is organized into five domains based on our review of the literature and consultation with experts in psychotherapy training and implementation/dissemination.

Download the Tool [PDF] Use the Interactive Tool

Applying the Tool to Evaluate Trainings

Organizations may offer multiple types of trainings. In this case, there are different ways to apply the tool:

Typical training
Complete the tool based on a typical training offered by the organization. Specifically, an organization could select a representative program and apply the tool based on that training. Although the results may not reflect all trainings offered by the organization, they would characterize a typical type of training for that organization.
Available capabilities
Score the tool based on the capabilities that an organization can offer as part of a training. Though each element may not be present in every training, scoring the tool in this way would provide a sense of the highest capacity the program can offer within each domain.
Individual training
Apply the tool to each individual training within an organization. For example, if a training is tailored for a specific audience or organization and has different features than a training for another audience/organization, the tool could be completed separately for each individual program. This would provide a more nuanced perspective on each specific training offered by the organization.

Scoring Guidance

Each item is rated on a scale from 1 to 5. This scale is anchored to the three categories of trainings, ranging from Raising Awareness (1) to Skill-Building (3) to Supporting Competence (5). Specific scoring anchors are provided to make the distinction between each level as concrete as possible.

As a general rule, all training leaders for a given training must include an element in delivering the training for that element to qualify as present in the training. For example, if one training leader typically provides guidance on measurement-based care but other training leaders do not mention or provide any resources on the topic, then the training should receive an item score of “1.” When sources of information about a training conflict (e.g., training slides versus report of training leader), raters should make an effort to resolve disagreement by consulting additional sources of information about the training.

The sections below provide specific guidance for each item on the scored portion of the TIP Tool. For each item, a definition is provided, as well as descriptions of any terms used in the scoring instructions. Specific guidance is then provided for coding each item, including a description of what constitutes a score at each level.

Training Program Identification

Information to complete this section can be collected from the training director or manager, website, or other written training materials.

Definition: This section is intended to record basic information on the training and details on how to gain additional information. Raters should indicate the date the tool was completed and by whom.

Training Characteristics

Information to complete this section can be collected from the training director or manager, website, or other written training materials.

Definition: This section is intended to record the focus of the training, including the target psychotherapies and diagnoses, the size of the training, and whether there are any minimum qualifications required for clinicians (e.g., licensure in a mental health field, any prerequisite course completions, etc.). The section also gathers information about the format of the training.

  • Training format refers to the way the training content is delivered. Raters should place a check box next to “in-person” and/or “web-based” if any elements of the training are conducted in person or online, respectively. If the training features both optional and required components of a particular format (in-person or web-based), the rater should place a check mark in both boxes.
  • Typical length of didactic training is the average number of hours required to complete the entire training. This number should include any required in-person and required web-based components, including training sessions, demonstrations, skills practice, discussion, or assessment. Optional components should not be included, such as reviewing supplemental materials, optional training components, time spent preparing for training sessions, or time spent on homework or outside skills practice. Supervision or consultation should also not be included.
  • Typical timing of didactic training refers to the spacing of required training sessions. If there is only one session, or if the training is typically conducted over several days without interruption, raters should select “One time, consecutive days.” If sessions take place over time, with breaks in between, raters should select “Spaced out over time.”

Scored Domains

The sections below provide more detail on the definition and scoring for each of the scored domains. While information to complete this section may be available in written materials, it may be necessary to gather the information directly from the training director, manager, or instructor.

Raters of each item must be familiar with the content of the training course and the way in which the training content is delivered. Raters should consider all recent (6–12 months) iterations of the training, including any variability caused by different training leaders or customizations made for trainees with different experience levels. Item scoring should account for this variability by endorsing the score that reflects how the training is typically delivered.

I. Didactic Training Format

IA. Length of Training

Definition: This is the total time required (in hours) to complete the entire didactic training (not including breaks).

  • This total should include the length of any required components of the training, including in-person or web-based training sessions and in-person or web-based demonstrations, skills practice, discussion, or assessment.
  • This total should not include time spent on any of the following activities related to the training: supervision or consultation, reviewing supplemental materials, optional training components, time spent preparing for training sessions, or time spent on homework or outside skills practice.
Item Scoring
  • Raising Awareness
    • (1) 2 hours or less. The training is two hours or less in total length, such as a one-time two-hour workshop or a self-directed web-based course that requires an estimated two hours to complete.
    • (2) 3 to 4 hours. The training requires approximately three or four hours to complete. Examples include a one-time four-hour workshop, a web-based facilitated training seminar completed in two sessions that are each two hours in length, or a self-directed web-based training that requires 3–4 hours to complete.
  • Skill-Building
    • (3) 5 to 7 hours. The training requires approximately five, six, or seven hours to complete. Examples include a day-long workshop, six weekly hour-long training sessions, or a web-based seminar that takes an estimated six hours to complete.
    • (4) 8 to 13 hours. The training is greater than or equal to eight hours and less than or equal to 13 hours in total length. Examples include two day-long workshops that are each six hours in length, ten weekly hour-long training sessions, or a web-based training that takes approximately eight to 12 hours to complete.
  • Supporting Competence
    • (5) 14+ hours. The training is 14 or more hours in total length. Examples include two day-long workshops that are each seven hours in length, ten weekly two-hour sessions, or a three-day workshop consisting of seven-hour training sessions.

IB. Level of Interactivity

Definition: This is the degree to which trainees interact with the training material through responding, reflecting, sharing, applying concepts, practicing skills, and otherwise engaging with the key concepts, theories, and skills featured in the required components of the didactic training. Interactivity may take place in person or online.

  • Self-guided trainings are facilitated by the trainee, and there is no training leader guiding either the in-person or web-based training session(s)—nor are there any trainee peers with whom to interact (either online or in person). Self-guided formats include self-directed web-based programs, computer-based trainings, and self-guided training through the use of a training manual or guide.
  • Didactic refers to the delivery of training content, such as theory or key concepts, through a lecture-style format in which the trainee is a listener and the training leader or web-based training program is the speaker or presenter of information.
  • Demonstrations may be in person, online, or in a video. Demonstrations include any of the following activities: live demonstrations by the trainer, video demonstrations of relevant concepts or skills, web-based audiovisual demonstrations, and and “fishbowl”-style role plays in which one or two trainees practice a skill and are observed by other trainees and/or the training leader.
    • A video showing a therapist introducing a treatment approach to a mock patient is an example of a video demonstration. A video introducing those concepts directly to the trainees is not a demonstration because there is no clinical example or demonstration.
  • Role play is an exercise in which most or all trainees practice (as the clinician) using the skills and concepts featured in the training by interacting with a mock or simulated therapy client, either in person or online. The mock client might be a fellow trainee or group of trainees, the training leader, or a simulation. Role-play exercises might take place in pairs or groups.
    • As discussed above, role plays conducted in “fishbowl” style in which the trainee(s) in the role play are observed by other trainees and/or the training leader do not count as role plays for this item. To qualify as a role play, most or all trainees must have the opportunity to practice the skills.
  • Other interactive elements (not demonstrations or role play) include
    • group discussion about interpretation, application, or understanding of training content, including discussion about a completed homework assignment
    • interactive quizzes or exercises (completed on paper, in person, or online) that require trainees to interact with the training content
    • games or activities (online or in person) in which trainees interact with one another to apply or learn the training material
    • contributions to peer discussion boards
    • case examples or vignettes presented by the training leader
    • skills practice during training (e.g., completing a thought record).
Item Scoring
  • Raising Awareness
    • (1) Self-guided. The trainee does not engage or interact directly with a training leader, nor does the trainee interact with trainee peers. Training content is delivered in a self-guided format, which may consist of a written or computer-based training manual, book, or online asynchronous web-based learning program that the trainee completes in his/her/their own time. An example of a training in this category is one that consists of a training manual plus quizzes on content.
    • (2) Didactic only. There is little to no interaction between the trainee and the training leader, and there is little to no interaction with other trainees. Training content is delivered by a training leader either in person or online. The trainee listens but does not interact with the leader or content apart from asking clarifying questions.
  • Skill-Building
    • (3) Didactic, plus EITHER demonstrations or other interactive elements. There is little to moderate interaction between the trainee and the training leader and/or trainee peers. Demonstrations (video or real-time) OR other interactive elements are used to show a real-world application of the training content. The program uses a lecture format (in person or online) in which trainees listen at times without interacting, and it also features demonstrations or applications of relevant concepts or skills. The training does not include role play.
    • (4) Didactic, plus BOTH demonstrations and other interactive elements. elements. There is moderate to significant interaction between the trainee and the training leader and/or trainee peers. The training uses a lecture format (in person or online) in which trainees listen at times without interacting, and it also features both demonstrations and applications of relevant concepts or skills. The training does not include role play.
  • Supporting Competence
    • (5) Didactic, plus ALL of the following: demonstrations, other interactive elements, and role plays. There is significant interaction between the trainee and the training leader and/or peers. The training uses a lecture format (in person or online) in which trainees listen at times without interacting, and it also features demonstrations, other applications of relevant concepts or skills, and role play(s).

IC. Individual Feedback During Training

Definition: This is information for the individual trainee conveyed during didactic training either verbally or in writing—and either in person or online—about individual learning, mastery, content knowledge, or skills pertaining to the training. Individual feedback is tailored in some way for the trainee; this item does not refer to feedback that might be given to the entire group of trainees.

  • Self-guided or self-generated feedback not tailored directly to an individual trainee, and it does not come from a person (e.g., trainer, peer). Examples include feedback generated from a self-assessment tool, tips identified through a training manual, and the results of a multiple-choice quiz.
  • Informal feedback from the trainer or peer(s) refers to information that is shared (verbally or in writing) in an ad hoc fashion with the trainee. Examples include
    • observations or comments reflecting on trainee skill practice or role play
    • peer feedback during small-group discussion
    • reflections on a skill practice exercise conducted in pairs
    • leader comments on a homework assignment.
  • Structured feedback uses a structured tool to provide feedback on strengths and opportunities for growth with regard to the course material. This feedback might pertain to performance (e.g., skills practice, role play) or to understanding of content. Examples include
    • graded or formal assessment on an assignment, exercise, or role play
    • use of a standard assessment to evaluate a role play or skill rehearsal.
  • If the rater is unable to observe the training, this item can be assessed by examining whether the materials (e.g., slide deck, handouts) reflect a clear intention to provide individual feedback, such as through the use of guidelines, scripts, checklists, or other tools.
Item Scoring
  • Raising Awareness
    • (1) None—or self-guided or self-generated only. No feedback is provided, or the feedback that is provided is the same for everyone, and it does not come from a person.
    • (2) Few trainees receive informal feedback from leader OR peer(s). The feedback conveyed to trainee(s) is informal, meaning that it is delivered in an ad hoc manner in response to the trainee. One, two, or three trainees receive feedback.
  • Skill-Building
    • (3) Some but not all trainees receive informal feedback from leader or peer(s). More than three but fewer than all trainees receive feedback. The feedback (verbal or written) conveyed to trainees is informal, meaning that it is delivered in an ad hoc manner in response to the trainee.
    • (4) All trainees receive informal feedback from leader OR peer(s). The feedback (verbal or written) conveyed to trainee(s) is informal, meaning that it is delivered in an ad hoc manner in response to the trainee. Trainings that score a 4 on this item likely create multiple different opportunities to deliver feedback so that, over time, every trainee has the opportunity to receive comments from the leader or peer(s). For example, role plays could include another trainee who serves as an observer and provides feedback following the role-play exercise.
  • Supporting Competence
    • (5) Structured, individual feedback. All trainees in the training receive individual feedback from the training leader(s), either verbal or written. The purpose of the feedback is to evaluate the trainee’s understanding or apparent mastery of training content by providing information about particular strengths or opportunities for improvement.

ID. Integration of Written Materials

Definition: This is the extent to which print resources are used and integrated in the training. This refers to review of the materials by trainees independently, outside of the didactic training (e.g., review of the materials before the training begins, between didactic training days for multi-day trainings, or after the training has ended).

  • Written materials include training manuals or guides, books, handouts generated from a book or manual, and handouts developed specifically for the training, as well as the availability of these materials online or through a computer program. This does not refer to printouts of PowerPoint slides used in the training but instead refers to supplemental resources.
Item Scoring
  • Raising Awareness
    • (1) No use of written materials. The training does not use or suggest that trainees use any materials, such as manuals, guides, books, or handouts. A training with this score does not offer any optional materials through an online platform or computer program.
    • (2) Written materials provided but not integrated into didactic training. The training may provide access to written materials, but this resource is not mentioned during training, nor is it suggested as a helpful prerequisite. Trainings with this score might list an optional text in a syllabus without mentioning it during the course, or they might post supplemental materials online for trainees to access based on interest.
  • Skill-Building
    • (3) Review of written materials suggested. The training suggests that trainees access written materials, but it does not require it. The trainer might reference the availability or possible use of materials during the training, but the materials are not discussed at length or in great detail.
    • (4) Review of written materials strongly encouraged but not required. The training encourages trainees to access written materials. Although use or review of the materials is not a requirement, there is a sincere attempt on the part of the training leader to encourage trainees to do so. For example, the leader might mention the availability of such resources more than once during the training, and there is likely an opportunity to discuss or review material related to these resources during the didactic training.
  • Supporting Competence
    • (5) Review of written materials required. Trainings meeting this score have a stated requirement involving a training manual, book, or other written resource that trainees are required to review or use as a part of the training. The requirement need not be enforced through an exam or other measure, but it must be a requirement of the training such that the majority of trainees do make use of these resources.

II. Didactic Training Content Area

IIA. Core Techniques and Theory

Definition: This is the extent to which the didactic training content includes and covers the theoretical underpinnings and central therapeutic techniques of the psychotherapy on which the training is focused. This item pertains to didactic training content only, not supervision/consultation.

  • Concepts refers to core techniques and/or theory:
    • Core techniques are the actual methods or processes by which a clinician delivers a psychotherapy. Examples of techniques include
      • in vivo exposure (facing a fear in a safe environment with the goal of decreasing future discomfort and avoidance associated with that fear)
      • breathing retraining (training to slow down breathing)
      • progressive muscle relaxation (an exercise in which muscles are squeezed and released in coordination with breathing to reduce tension)
      • identifying cognitive distortions (inaccurate, incomplete, or unbalanced thoughts that have a negative impact on the patient’s mood or experience)
      • cognitive restructuring (a process of evaluating thoughts to examine them and shift thoughts to be more accurate, complete, and balanced)
      • behavioral experiments (engaging in activities to test beliefs or perceptions)
      • activity monitoring/scheduling (identifying and increasing the frequency of behaviors that improve mood)
      • using a Subjective Units of Distress Scale.
    • Theory is an explanation of how the psychotherapy helps patients change. For example, a training that covers the theory of cognitive behavioral therapy would explain how thoughts and behaviors relate to mood. Other examples of theory include
      • cognitive theory of posttraumatic stress disorder, which suggests that the way an individual recalls traumatic events can shape subsequent adaptation and well-being
      • emotional processing theory, which holds that trauma and memory are interrelated such that trauma memories must be reprocessed to assign new meaning to reduce posttraumatic stress disorder symptoms
  • Supplemental materials include training manuals or guides; books; course handouts; such resources as videos, graphics, websites, or reports that are posted to a website or available through a computer program; and anything else that is provided to trainees for review outside of the didactic training.
Item Scoring
  • Raising Awareness
    • (1) Not included. The training does not include any of the core techniques or theory during didactic training, nor does it provide any resources on these topics.
    • (2) Concepts addressed in supplemental materials or resources but not discussed in didactic training. The training provides materials or resources that mention core techniques and theory, but they are not discussed during didactic training.
  • Skill-Building
    • (3) Concepts introduced in didactic training. At least one core technique and at least one aspect of theory is explicitly mentioned during didactic training without going into depth with examples. These components might only be mentioned once, or they might be mentioned more than once without any applied examples. Trainees do not have the opportunity to practice or apply information on this topic.
    • (4) Concepts discussed with examples in didactic training.. Technique and theory are introduced and explicitly discussed with applied examples for either or both components during didactic training. There are no opportunities to practice the techniques or apply concepts of theory through role play or any other applied method.
  • Supporting Competence
    • (5) Trainees given opportunity to practice or apply concepts in didactic training. Trainees given opportunity to practice or apply concepts in didactic training. The training provides explicit instruction on both core techniques and theory during didactic training, including applied examples. Trainees are given the opportunity to practice or apply the concepts that are discussed. Practice might take the form of a role play or skill exercise, or trainees might discuss hypothetical scenarios and the way they would respond in small groups. In general, the examples and the exploration of the material that are provided during training are relatively in depth and detailed. The training does not necessarily cover all aspects of theory and techniques for the psychotherapy of focus, but it likely touches on more than one aspect of each.

IIB. Clinical Judgment and Decisionmaking

Definition: This is the extent to which the didactic training content includes instruction about clinical decisionmaking with the psychotherapy of focus, or how to decide what to do during a psychotherapy session. This item pertains to didactic training content only, not supervision/consultation.

  • Examples of clinical decisionmaking (concepts and examples) include instruction on
    • what to do when the basics do not work
    • how to decide when or how to use a therapy technique
    • an overview of common therapist mistakes implementing the therapy and guidance on how to avoid making them
    • examples of ways the clinician can maintain adherence to the psychotherapy while tailoring its application to the needs of the patient.
    • examples of modifications to a therapy that can be made so that the therapy is more culturally competent or otherwise tailored to a particular population without deviating from the core theory.
  • Supplemental materials include training manuals or guides; books; course handouts; such resources as videos, graphics, websites, or reports that are posted to a website or available through a computer program; and anything else that is provided to trainees for review outside of the didactic training period.
Item Scoring
  • Raising Awareness
    • (1) Not included. The training does not cover the topic of clinical judgment and decisionmaking during didactic training, nor does it provide any resources on this topic.
    • (2) Concepts addressed in supplemental materials or resources but not discussed in didactic training. The training provides materials or resources that mention the concept of clinical judgment and decisionmaking using the evidence-based practice of focus, but this topic is neither discussed nor practiced during didactic training.
  • Skill-Building
    • (3) Concepts introduced in didactic training. The training explicitly mentions the concept of clinical judgment and decisionmaking during didactic training without going into depth with examples. Trainees do not have the opportunity to practice or apply information on this topic.
    • (4) Concepts discussed with clinical examples in didactic training.. Clinical decisionmaking is introduced and explicitly discussed with applied examples during the didactic training. There are no opportunities for trainees to practice or apply information on this topic.
  • Supporting Competence
    • (5) Trainees given opportunity to practice or apply concepts in didactic training. The training provides explicit instruction on clinical decisionmaking during didactic training, including applied examples. Trainees are given the opportunity to practice or apply the concepts that are discussed. Practice might take the form of a role play or skill exercise, or trainees might discuss in small groups a series of hypothetical scenarios and the way they might respond based on the information provided during the training.

IIC. Addressing Co-Occurring Disorders

Definition: This is the extent to which the didactic training content includes instruction about addressing co-occurring disorders with the psychotherapy of focus. If the treatment does not explicitly include techniques for treatment of co-occurring disorders, the training could include an overview of how to manage patients who present with co-occurring disorders (e.g., through supplemental treatment approaches or referrals). This item pertains to didactic training content only, not supervision/consultation.

  • Supplemental materials include training manuals or guides; books; course handouts; such resources as videos, graphics, websites, or reports that are posted to a website or available through a computer program; and anything else that is provided to trainees for review outside of the didactic training period.
Item Scoring
  • Raising Awareness
    • (1) Not included. The training does not include the topic of co-occurring disorders during didactic training, nor does it provide any resources on this topic.
    • (2) Concept addressed in supplemental materials or resources but not discussed in didactic training. The training provides materials or resources that mention the concept of treating co-occurring disorders with the psychotherapy of focus, but this topic is neither discussed nor practiced during didactic training.
  • Skill-Building
    • (3) Concept introduced in didactic training. The training explicitly mentions the concept of addressing co-occurring disorders with the psychotherapy of focus during didactic training without going into depth with examples. Trainees do not have the opportunity to practice or apply information on this topic.
    • (4) Concept discussed with clinical examples in didactic training. Addressing co-occurring disorders with the psychotherapy of focus is explicitly discussed with applied examples during didactic training. There are no opportunities for trainees to practice or apply information on this topic.
  • Supporting Competence
    • (5) Trainees given opportunity to practice or apply concept in didactic training. The training provides explicit instruction on addressing co-occurring disorders with the psychotherapy of focus during didactic training, including applied examples. Trainees are given the opportunity to practice or apply the concepts that are discussed. Practice might take the form of a role play or skill exercise, or trainees might discuss in small groups a hypothetical case formulation and the way they might treat or manage that client differently than someone without co-occurring conditions, based on the information provided during the training.

IID. Measurement-Based Care

Definition: This is the extent to which the didactic training content includes the topic of measurement-based care. Measurement-based care is the systematic, repeated administration of symptom measures to assess patient progress over the course of treatment and use of the information from these measures to inform clinical decisionmaking at the individual patient level (Fortney et al., 2016). Common symptom measures include the Patient Health Questionnaire-9; the Posttraumatic Stress Disorder Checklist for the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition; and the Columbia Suicide Severity Rating Scale. This item pertains to didactic training content only, not supervision/consultation.

  • Supplemental materials include training manuals or guides; books; course handouts; such resources as videos, graphics, websites, or reports that are posted to a website or available through a computer program; and anything else that is provided to trainees for review outside of the didactic training period.
Item Scoring
  • Raising Awareness
    • (1) Not included. The training does not include the topic of measurement-based care during didactic training, nor does it provide any resources on this topic.
    • (2) Concept addressed in supplemental materials or resources but not discussed in didactic training. The training provides materials or resources that mention the concept of measurement-based care, but this topic is neither discussed nor practiced during didactic training.
  • Skill-Building
    • (3) Concept introduced in didactic training. The training explicitly mentions the concept of measurement-based care during didactic training without going into depth with examples. Trainees do not have the opportunity to practice or apply information on this topic.
    • (4) Concept discussed with clinical examples in didactic training.. Measurement-based care is explicitly discussed with applied examples during didactic training. There are no opportunities for trainees to practice or apply information on this topic.
  • Supporting Competence
    • (5) Trainees given opportunity to practice or apply concept in didactic training. The training provides explicit instruction on measurement-based care during didactic training (e.g., selecting appropriate measures, frequency of administering measures, how to use the data to adjust treatment), including applied examples. Trainees are given the opportunity to practice or apply the concepts that are discussed. Practice might take the form of a role play or skill exercise, or trainees might discuss in small groups a hypothetical case and strategies to integrate measurement-based care based on the information provided during the training (e.g., ways to discuss treatment progress with patients using measurement-based care data).

III. Consultation and Supervision

IIIA. Availability of Consultation/Supervision

Definition: This is the extent to which the training offers consultation or supervision. Supervision or consultation involves an expert in the modality providing support, guidance, and critical feedback to the trainee over time. Supervision/consultation may be conducted in person or via telephone or video/web conference and as group or individual supervision.

Item Scoring
  • Raising Awareness
    • (1) Not referenced. The training does not mention supervision during didactic training.
    • (2) Consultation/supervision (from training or training partner) is referenced during didactic training. The training leader might mention consultation/supervision verbally, or an online training might contain a written reference to consultation/supervision, but trainees are not prompted to consider whether they might want to seek consultation/supervision.
  • Skill-Building
    • (3) Consultation/supervision (from training or training partner) is available and is suggested. The training suggests that trainees might seek consultation/supervision, either provided by the training or through a training partner. The training does not try to persuade trainees to seek consultation/supervision, but it is referenced as an option.
    • (4) Consultation/supervision (from training or training partner) is available and is encouraged. The training attempts to persuade trainees to seek consultation/supervision, but it is not a requirement.
  • Supporting Competence
    • (5) Consultation/supervision (from training or training partner) is available and is required. Every trainee in the program is required to complete consultation/supervision.

IIIB. Review of Sessions

Definition: This is encouragement or requirement for the review of therapy sessions as part of the consultation or supervision component of the training.

  • Consultation/supervision involves an expert in the psychotherapy providing support, guidance, and critical feedback to the trainee over time. It may be conducted in person, via telephone, or via video/web conference and as a group or individually.
  • Trainee therapy sessions are those sessions conducted by the trainee with an actual client. Recordings or reports of practice sessions with mock or simulated patients are not to be included in rater assessment and scoring of this item.
Item Scoring
  • Raising Awareness
    • (1) Not referenced. The training does not offer review of recorded therapy sessions by a clinical supervisor, nor does it suggest or encourage trainees to seek outside supervision for review of recorded therapy sessions.
    • (2) Formal review of trainee therapy sessions is not provided but is suggested. The training encourages trainees to seek outside supervision for review of recorded or therapy sessions.
  • Skill-Building
    • (3) Written or verbal self-report of trainee therapy session(s) reviewed. The training provides review of written or verbal self-report of trainee therapy session(s) as a part of the training. This review process can take place in person, over the phone, or by videoconference. The supervisor might provide a written or verbal evaluation of the trainee or may use a standardized tool to provide feedback to the trainee, but an evaluative component is not required to receive a score of 3.
    • (4) Live or recorded trainee therapy session(s) reviewed and evaluated without a structured coding tool. The training provides review of live or recorded trainee therapy session(s), and this review is accompanied by an unstructured evaluative component.
  • Supporting Competence
    • (5) Live or recorded therapy session(s) reviewed and evaluated using a structured coding tool. The training provides an evaluation of live or recorded trainee therapy session(s), giving feedback to trainees using a structured coding tool.

IIIC. Supervision Length

Definition: This is the provision of trainee supervision or consultation as a component of training. Supervision or consultation could include assessment of skills using a structured coding tool of trainee adherence and competence.

  • Consultation/supervision involves an expert in the psychotherapy providing support, guidance, and critical feedback to the trainee over time. It may be conducted in person, via telephone, or video/web conference and as a group or individually.
  • Structured coding tools are standardized instruments designed to assess adherence and competence. These include validated measures, such as the Cognitive Therapy Scale—Revised (Blackburn et al., 2001) or the Yale Adherence and Competence Scale (Carroll et al., 2000). These could also include checklists developed to assess the absence or presence of key elements of the psychotherapy or rating scales developed to assess adherence and competence that have not been formally validated.
Item Scoring
  • Raising Awareness
    • (1) No consultation/supervision provided. Consultation/supervision might be mentioned, suggested, or even encouraged by the training, but it is not a requirement.
    • (2) Supervision/consultation provided, but no specified length or case target. Consultation/supervision is provided on multiple occasions over time, but there is no minimum length of time or number of client cases required for consultation/supervision.
  • Skill-Building
    • (3) Supervision/consultation provided for at least two months. The training requires a minimum of two months of consultation/supervision, either provided by a partner training or facilitated by the training itself.
    • (4) Supervision/consultation provided for at least six months or until trainee demonstrates adherence and competence (assessed using a structured coding tool) with one patient. The training requires a minimum of six months of consultation/supervision, either provided by the training organization or facilitated by the training itself, unless the trainee demonstrates adherence and competence before six months. Assessment of trainee adherence and competence must be completed using a structured coding tool to assess trainee live or recorded therapy sessions. Though some trainings may require completion of a certain number of training cases, for trainings offering less than six months of supervision to obtain this score, there must be assessment of adherence and competence with those training cases.
  • Supporting Competence
    • (5) Supervision/consultation provided and continues until trainee demonstrates adherence and competence (assessed using a structured coding tool) in session with at least two patients. The training requires that trainees continue in consultation/supervision until they achieve adherence and competence with two patients, based on an assessment of trainee live or recorded therapy sessions using a structured coding tool.

IIID. Certification

Definition: This is encouragement or requirement of trainee certification1 as a part of the training.

  • Certification in a psychotherapy typically indicates that the clinician has demonstrated an ability to deliver the psychotherapy with fidelity. An example of a certification program is the Academy of Cognitive Therapy, which certifies clinicians in cognitive behavioral therapy. Other examples include network provider status for cognitive processing therapy or prolonged exposure. These are distinct from licensure and other certification or credentialing programs that do not include an assessment of fidelity to the psychotherapy. This does not refer to certificates of completion provided at the end of a training and must involve some demonstration of skill or fidelity (e.g., submitting progress notes from sample patients, submitting treatment recordings).

1 It is important to acknowledge that not all evidence-based psychotherapies have an associated certification process or accrediting body. Moreover, there can be substantial variability in the requirements when certification is an option. That said, the types of certification processes assessed by this item are a way to emphasize the importance of formally testing a clinician’s skill. In this way, encouraging trainees to obtain certification may provide an incentive for clinicians to reach adherence and competence.

Item Scoring
  • Raising Awareness
    • (1) Training does not mention certification to trainees. A psychotherapy that does not have an established certification process would receive a score of “1” for this item.
    • (2) Supplemental materials or resources mention certification but not discussed in didactic training. Supplemental materials might include handouts or online resources.
  • Skill-Building
    • (3) Certification is mentioned during training or supervision as an option. Trainees are neither required nor encouraged by the training to pursue certification.
    • (4) Certification is encouraged. Certification is mentioned during didactic training or supervision as an option, and trainees are encouraged to think about obtaining certification upon completion of the training.
  • Supporting Competence
    • (5) Certification is strongly encouraged, and training tracks the number of trainees who receive certification. Certification is the ultimate goal of the training. While not all trainees may be certified, it is strongly encouraged, and the training tracks the number of trainees who receive certification.

IV. Program Evaluation

IVA. Course Evaluation

Definition: This is an assessment of the trainee experience and satisfaction with the training, and the extent to which any information collected is used to improve the training.

Item Scoring
  • Raising Awareness
    • (1) No course evaluation of the training. The training does not provide opportunity for trainees to give any formal feedback or course evaluation.
    • (2) Optional course evaluation or other opportunity to share trainee feedback with the training. The training leader or training staff express a willingness to receive feedback, either informally or through a course evaluation. The training program does not require the training leader(s) to collect trainee feedback.
  • Skill-Building
    • (3) Required assessment of trainee experience with the training. The training requires training leader(s) to collect trainee feedback, but the information is not routinely shared with trainers, nor is it used to improve the training.
    • (4) Information from the required assessment of trainee experience with the training is routinely shared with trainers. The training requires training leader(s) to collect trainee feedback. Although the training leaders are informed of trainee feedback, this information is not otherwise used to make significant changes to the content or delivery of the training.
  • Supporting Competence
    • (5) Information from the required assessment of trainee experience with training is used to improve training content, format, and facilitation. The training requires training leader(s) to collect trainee feedback. Not only is the information about trainee feedback routinely shared with training leaders, but the training also has responded by making significant improvements to the content or delivery of training in response to trainee feedback.

IVB. Knowledge

Definition: This is an assessment of trainee knowledge related to the psychotherapy of focus.

  • An assessment of perceived knowledge asks the trainee to estimate how much they have learned about the psychotherapy (e.g., participant ratings of the extent to which participation in the training led to an increase in knowledge). Perceived knowledge tests are coded by the training to determine the trainee’s self-reported perceived level of knowledge about the psychotherapy.
  • A knowledge test assesses understanding of key concepts and techniques. Unlike tests of perceived knowledge, tests of knowledge are graded by the training to determine the trainee’s actual, objective level of knowledge about the psychotherapy.
Item Scoring
  • Raising Awareness
    • (1) No assessment of trainee knowledge.
    • (2) Brief post-training assessment (5 items or fewer) of trainee-perceived knowledge change. There may or may not be a pre-course assessment of perceived trainee knowledge.
  • Skill-Building
    • (3) Post-training assessment of trainee-perceived knowledge change. The training asks trainees to reflect on whether their knowledge about the psychotherapy has changed. There may or may not be a pre-course assessment of perceived trainee knowledge (e.g., "How knowledgeable are you about creating thought records?"). The training does not administer a knowledge test (e.g., "Which of the following is a definition of an automatic thought?").
    • (4) Post-training test of trainee knowledge. The training may or may not ask trainees about perceived knowledge, but it must assess knowledge using a test that is scored by the training leader.
  • Supporting Competence
    • (5) Pre- and post-training test of trainee knowledge. Trainees complete a test related to the training content both before and after the training so that pre/post scores can be compared to determine change over time. The training may or may not ask trainees about perceived knowledge, but it must assess knowledge using a test that is scored by the training leader.

IVC. Skills

Definition: This is an assessment of trainee skills related to the psychotherapy of focus.

Item Scoring
  • Raising Awareness
    • (1) No assessment of trainee skills. The training does not provide any assessment of trainee skills related to the psychotherapy of focus.
    • (2) Post-training assessment of trainee-perceived therapeutic skills included in training. The training asks the trainee to indicate which therapeutic skills were included in the training content. These questions might be incorporated in a questionnaire about satisfaction with the training or asked independently as a measure of what skills trainees recall learning during the training. The training does not ask the trainees about their perceived ability to use the therapeutic skill(s).
  • Skill-Building
    • (3) Post-training assessment of trainee-perceived ability to use specific therapeutic skills. The training asks the trainee to report their perceived ability to use therapeutic skills associated with the psychotherapy of focus. For example, a post-training assessment for a cognitive processing therapy training might ask trainees if they know how to work with a client to identify stuck points. The training does not ask the trainees about how confident they are in their ability to use the therapeutic skill(s).
    • (4) Post-training assessment of trainee self-reported confidence in applying specific therapeutic skills. The training asks trainees to estimate how well they think they might do with applying therapeutic skill(s) during psychotherapy with a client.
  • Supporting Competence
    • (5) Post-training assessment of trainee ability to use the specific therapeutic skills (e.g., role play). Rather than asking the trainee to report on their perceived learning, ability, or confidence, the training uses an objective test to evaluate trainee therapeutic skill that is facilitated through a role play or similar exercise.

V. Implementation Facilitation

VA. Development of a Plan for Implementation

Definition: These are activities during didactic training to assist clinicians in developing an implementation plan for the incorporation of new knowledge and skills into their clinical practice. These activities are distinct from the specific knowledge and skills associated with the psychotherapy; instead, they focus on contextual factors that may shape a trainee’s ability to deliver the psychotherapy in their usual care setting. For example, this may include organizational context (e.g., leadership support), clinician attitudes or perceptions of the treatment, and patient attitudes or perceptions.

  • Implementation plan refers to a written document that will guide the implementation of the psychotherapy in a trainee’s practice setting. It may include elements such as next steps to prepare for implementation (e.g., plans for identifying appropriate patients, securing supervision), resources needed, identification of barriers and potential solutions, and identification of indicators of successful implementation.
Item Scoring
  • Raising Awareness
    • (1) Not present. There is no mention of implementation barriers or facilitators and no mention of developing a plan for implementing the psychotherapy into the trainee’s clinical practice.
    • (2) Didactic training indicates that trainees should consider implementation context. This includes implementation barriers and/or facilitators. This indication could be made through written materials provided during the didactic training or verbally during the didactic training. However, the training does not include any formal discussion of implementation and does not provide specific guidance as to how to develop an implementation plan.
  • Skill-Building
    • (3) Didactic training includes discussion of implementation context. This discussion might include common implementation challenges or ways to address these challenges. However, the discussion does not include consideration of barriers and facilitators specific to each individual trainee’s practice setting. Trainees might have access to a template that could be used to develop an implementation plan that they could complete on their own, but there is no facilitated completion of such a plan during didactic training.
    • (4) Training provides time for clinicians to develop a basic plan for implementation. In addition to discussion of implementation context, the training provides time for trainees to begin developing a basic plan for implementation. For example, trainees may be asked to identify the three implementation barriers most relevant to their organization and generate specific solutions.
  • Supporting Competence
    • (5) Training includes the development of a comprehensive plan for implementation. The training provides time for trainees to develop a comprehensive plan for implementation. Trainees are guided to consider any challenges they may encounter and ways to address those issues. This could include a structured assessment of implementation barriers that is completed by the trainee or their organization, such as a measure of organizational context or clinician attitudes. Trainees are provided with a structured way to develop the implementation plan, such as a worksheet or implementation plan template.

VB. Implementation Support

Definition: This is the amount of support for implementation of the psychotherapy that is available through the training, outside of implementation support provided during the didactic component of the training. Implementation support includes working with a trainee and/or the trainee’s organization to identify and address implementation challenges. It may also include implementation support provided to trainees during consultation/supervision.

  • Outside resources include any of the following:
    • websites or links to information online
    • recommended training manuals, guides, books, or other written materials
    • a list of recommended agencies or institutions that support implementation.
Item Scoring
  • Raising Awareness
    • (1) Not available. There is no support available for implementation.
    • (2) Implementation support is not available, but self-guided use of outside resources is suggested. The training identifies specific resources that trainees might use to support implementation.
  • Skill-Building
    • (3) Informal implementation consultation is available from the training. The training provides limited support for implementation after the didactic component of the training. For example, implementation challenges may be addressed as part of consultation following a didactic training if a trainee raises a question about implementation challenges (e.g., difficulty identifying patients), but implementation support is not a standard topic during consultation.
    • (4) Formal implementation support is available before or after didactic training. Implementation support is a standardized element of the training. This might include brief consultation with the trainee’s agency to understand organizational context and make recommendations or addressing implementation barriers as a standardized component of post-didactic training consultation sessions. The training provides support for implementation at one time point only: before or after didactic training.
  • Supporting Competence
    • (5) Formal implementation support is available from the training before and after didactic training. . Implementation support is a standardized element of the training, and the training provides support for implementation before and after the didactic training.

Final Calculation

Scores on individual items should be selected based on the best available information about the training.

TIP Score Calculation

The total sum is calculated by summing the raw item scores for the 17 items in each of the five domain sections I through V:

  • Training Format (items A–D),
  • Training Content Areas (items A–D),
  • Consultation and Supervision (items A–D),
  • Program Evaluation (items A–C), and
  • Implementation Facilitation (items A–B).

The total sum will range between 17 and 85.

The TIP score is calculated by dividing the total sum by 17. TIP scores will range between 1 and 5, or between Raising Awareness (1) and Supporting Competence (5). The overall TIP total score should be used to determine the corresponding TIP score category (Raising Awareness, Skill-Building, or Supporting Competence). A TIP total score corresponds to each category as follows:

  • Raising Awareness - a total score of 1.00-2.33
  • Skill-Building - a total score of 2.34-3.66, and
  • Supporting Competence - a total score of 3.67-5.00

Domain Sum Ranges and Scores

The domain sums and scores in each section may provide more-specific feedback about the strengths and opportunities for program changes. Each domain sum is calculated by summing the raw item scores for the items within that domain. Domain sums will range between 4 and 20 for Didactic Training Format, Didactic Training Content Areas, and Attributes of Consultation/Supervision. The domain sum for Program Evaluation will range between 3 and 15, and the section sum for Implementation Facilitation will range between 2 and 10. The more informative TIP domain score is calculated by dividing the domain sum by the total number of items in that section, as specified on the scoring sheet. TIP domain scores will range between 1 and 5, or between Raising Awareness (1) and Supporting Competence (5). Each domain score should be used to determine the corresponding domain score category (Raising Awareness, Skill-Building, or Supporting Competence). A domain score of 1.00-2.33 corresponds to Raising Awareness, a domain score of 2.34-3.66 corresponds to Skill-Building, and a domain score of 3.67-5.00 corresponds to Supporting Competence.

After calculating the domain and overall scores, the results can be plotted using the interactive TIP Tool. Alternatively, a blank TIP Tool graph can also be downloaded [PDF]. Plotting the results can assist in the interpretation of TIP Tool scores.

The TIP Tool Continuum

The TIP Tool continuum reflects the need for different training options and allows better understanding of where a psychotherapy training falls on this continuum.

  • Raising Awareness

    Provides an overview or introduction to a given psychotherapy.

  • Skill-Building

    Promotes change in clinical practice by providing the foundation for developing skills and therapeutic techniques.

  • Supporting Competence

    Develops skills consistent with a given psychotherapy and helps clinicians achieve adherence and competence.

Learn More