Accountability, Reporting, and CTE Data

Perkins and Core Performance Indicator Definitions

Perkins V provides a list of secondary and postsecondary core performance indicators. As part of the state plan development process, states were directed to establish how these indicators would be measured and how the populations measured (CTE participants and concentrators) would be defined. Below are the operational secondary and postsecondary definitions for CTE students and the core indicators that will be used to report data for accountability purposes. 

 

Secondary CTE Definitions

Participant: Any 9th-12th grade student who successfully completes one or more courses which are part of a state- approved secondary CTE program. 

Concentrator: Any 9th-12th-grade student who successfully completes 150+ course hours (or more), which are part of a state-approved secondary CTE program, within one career field. 

 

Secondary Core Accountability Indicator Definitions

1S1: FOUR-YEAR COHORT GRADUATION RATES 

Statute Definition: The percentage of CTE concentrators who graduate high school, as measured by the four-year adjusted cohort graduation rate defined in section 8101 of the Elementary and Secondary Education Act of 1965 (ESEA). 

Operational  Definition 

Numerator: Number of CTE concentrators who, in the reporting year were included as graduated in the state’s computation of its four-year cohort graduation rate. 

Denominator: Number of CTE concentrators who, in the reporting year, were included in the state’s four-year computation of its four-year cohort graduation rate. Includes: 1) graduates, 2) dropouts, 3) continuing, and 4) unknown. 

2S1: ACADEMIC PROFICIENCY: READING/LANGUAGE ARTS 

Statute Definition: CTE concentrator proficiency in the challenging state academic standards adopted by the state under section 1111(b)(1) of ESEA, as measured by the academic assessments in reading/language arts as described in section 1111(b)(2) of such Act. 

 

Operational  Definition 

Numerator: Number of 10th grade CTE concentrators who met or exceeded the proficiency level on the statewide high school reading assessments, either the Minnesota Comprehensive Assessment (MCA) or Minnesota Test of Academic Skills (MTAS). 

Denominator: Number of students who were CTE concentrators by the end of their 10th-grade year who took the MCA or MTAS and whose high school reading assessment scores were included in the state’s ESSA computation. 

 

2S2: ACADEMIC PROFICIENCY: MATHEMATICS 

Statute Definition: CTE concentrator proficiency in the challenging state academic standards adopted by the state under section 1111(b)(1) of ESEA as measured by the academic assessments in mathematics as described in section 1111(b)(2) of such Act. 

Operational  Definition 

Numerator: Number of 11th grade CTE concentrators who met or exceeded the proficiency level on the statewide high school mathematics assessments, either the MCA or MTAS. 

Denominator: Number of students who were CTE concentrators by the end of their 11th grade year, who took the MCA or MTAS and whose high school mathematics assessment scores were included in the state’s ESSA computation. 

 

3S1: POST-PROGRAM PLACEMENT 

Statute Definition: The percentage of CTE concentrators who, in the second quarter after exiting from secondary education, are in postsecondary education or advanced training, military service or a service program that receives assistance under title I of the National and Community Service Act of 1990 (42 U.S.C. 12511 et seq.), are volunteers as described in section 5(a) of the Peace Corps Act (22 U.S.C. 2504(a)), or are employed. 

Operational  Definition 

Numerator: Number of CTE concentrators who, in the second quarter after graduating high school, are in postsecondary education or advanced training, military service or a service program that receives assistance under title I of the National and Community Service Act of 1990 (42 U.S.C. 12511 et seq.), are volunteers as described in section 

5(a) of the Peace Corps Act (22 U.S.C. 2504(a)), or are employed. 

Denominator: Number of CTE concentrators who graduated high school. 

4S1: NONTRADITIONAL PROGRAM CONCENTRATION 

Statute Definition: The percentage of CTE concentrators in career and technical education programs and programs of study that lead to non-traditional fields. 

 

Operational  Definition 

Numerator: Number of CTE concentrators from underrepresented gender groups who completed a program that leads to employment in nontraditional fields during the reporting year. 

Denominator: Number of CTE concentrators who completed a program that leads to employment in nontraditional fields during the reporting year. 

 

5S3: PROGRAM QUALITY: WORK-BASED LEARNING 

Statute Definition: The percentage of CTE concentrators graduating from high school having participated in work-based learning. 

 

Operational  Definition 

Numerator: Number of CTE concentrators who successfully completed one or more work-based learning courses prior to graduation. 

Denominator: Number of CTE concentrators who graduated high school. 

 

Postsecondary CTE Definitions

Participant: A student enrolled in a Minnesota State two-year college who belongs to a particular CTE entering cohort and who: 

• Earned more than zero college-level credits (cumulative) AND, within the cohort timeframe, was enrolled in a CTE program/award level or, 
• Earned more than zero college level credits in CTE course(s) 

Concentrator: A student enrolled in a Minnesota State two- year college who belongs to a particular CTE entering cohort and who: 

• Is enrolled in a long-term (12 or more credits) CTE program/award level AND earned 12 or more college level credits (cumulative), or 
• Completed a CTE award in a short-term (less than 12 credit) CTE program within the cohort timeframe. 

(Students enrolled in a short-term program who have not yet received that award are counted as participants. In addition, students who meet concentrator criteria but then receive only a non- CTE award within the cohort timeframe are counted as participants.) 

 

Postsecondary Core Accountability Indicator Definitions

 

1P1: POSTSECONDARY RETENTION AND PLACEMENT 

Statute Definition: The percentage of CTE concentrators who, during the second quarter after program completion, remain enrolled in postsecondary education, are in advanced training, military service, or a service program that receives assistance under title I of the National And Community Service Act of 1990 (42 U.S.C. 12511 et seq.), are volunteers as described in section 5(a) of the Peace Corps Act (22 U.S.C. 2504(a)), or are placed or retained in employment. 

Operational  Definition 

Numerator: Number of CTE concentrators in the cohort who, during the second quarter following program completion, are one or more of the following: enrolled in 

postsecondary education; in advanced training; in military service; in national or community service or the Peace Corps; or employed. 

Denominator: Number of CTE concentrators who complete a CTE program/award (i.e., certificate, diploma, AAS or AS) within the cohort timeframe. 

 

2P1: EARNED RECOGNIZED POSTSECONDARY CREDENTIAL 

Statute Definition: The percentage of CTE concentrators who receive a recognized postsecondary credential during participation in or within 1 Year of program completion. 

 

Operational  Definition 

Numerator: Number of CTE concentrators who complete a CTE program/award (i.e., receive a CTE certificate, diploma, AAS or AS) within cohort timeframe. 

Denominator: Number of CTE concentrators in the cohort. 

 

3P1: NONTRADITIONAL PROGRAM ENROLLMENT 

Statute Definition: The percentage of CTE concentrators in career and technical education programs and programs of study that lead to non-traditional fields. 

 

Operational  Definition 

Numerator: Number of CTE concentrators in the cohort who are enrolled in a CTE program classified as nontraditional for their gender. 

Denominator: Number of CTE concentrators in the cohort enrolled in a CTE program classified as nontraditional. 

While CTE “completers” are not defined in the Perkins V legislation, core indicators 1P1 and 2P1 both include that population of students. Minnesota defines CTE completers as students enrolled in a Minnesota State two-year college who belong to a particular CTE entering cohort and who receive a CTE award in a long- or short-term Perkins-eligible program within the cohort timeframe. In addition, Minnesota defines a CTE award as an AAS, AS, certificate or diploma. 

 

State-Determined and Local Levels of Performance

State-Determined Levels of Performance

Perkins V legislation requires eligible agencies (i.e., the state) to establish state-determined levels of performance for all four grant years and include them in the state plan submitted to the OCTAE in the spring of 2020. 

State performance levels were determined using a process that involved eligible recipient and stakeholder input. They are required to show continuous improvement in the performance of CTE concentrators and were subject to a 60-day public comment process prior to being submitted in the state plan. 

There is no negotiation process regarding levels of performance between the state and OCTAE. The state-determined performance levels, along with the methodology and process used to set them, were submitted in the state plan and were accepted by OCTAE. The state may revise indicator performance levels, but depending on the circumstances (unanticipated circumstances or other reasons), the requirements for the process and criteria that must be followed in making those adjustments are different. 

The state method for establishing state and local performance levels involves using a factor of the standard deviation of historical data for each performance indicator to establish what would be "meaningful progress" on that indicator by the fourth year of the grant. 

The state will be evaluated on core indicator performance against the appropriate state-determined performance level for each indicator every December. 

 

Local Levels of Performance

Similar to the state-determined levels of performance, local levels of performance were established for all four grant years. 

Local consortium annual levels of performance were determined by looking at the opportunity for improvement at the state level and seeing what portion of that improvement is held by each consortium. The amount of improvement is apportioned to consortia, using the same rate of annual growth that was used to establish the state annual levels of performance for that indicator. 

Unlike under Perkins IV requirements, there will be no annual negotiation process to determine local consortium performance indicator levels. Instead, consortia may submit requested changes to their levels of performance in their APR. However, if the consortium or the state is on an improvement plan for an indicator, performance levels for that indicator cannot be adjusted. Consortia are not required to suggest a specific change to the numerical value, but rather they must provide sufficient rationale to justify deviation from the standard methodology. Consortia may request changes to performance levels for any indicator for any grant year(s). Once received, state staff will consider the rationale provided in the APR and determine whether or how much to adjust the consortium performance level and the state level if needed. If the revised consortium level of performance impacts the overall state level of performance for an indicator, the adjusted state performance level is required to undergo 60 days of public comment. Please see the “Adjusting Consortium Levels of Performance” document on the CTE Consortium Resources site for additional details.

Consortia will be evaluated on their core indicator performance for the previous reporting year in the spring of each year, shortly following the submission of the state CAR. For example, consortia would be evaluated on reporting year 2020 in early spring 2021. 

 

State Consolidated Annual Report (CAR)

Each January, Perkins V requires the state to submit the CAR to OCTAE. The CAR includes the state’s progress on meeting state-determined performance levels from the most recently completed reporting year. Data regarding performance for each core indicator, and for each of the subgroups of students, special populations, and CTE clusters must be reported for each indicator. 

CTE participant and concentrator counts disaggregated by subgroups of students and special populations by CTE Cluster must also be reported. The disaggregation of the data is not required if the number of students would be considered private data or potentially reveal personally identifiable data about a student. 

If the state does not meet at least 90 percent of the state- determined performance level for any of the core indicators of performance for all CTE concentrators, the state will be required to implement an improvement plan during the subsequent year. The improvement plan must include an analysis of the gaps in performance among the special population and other disaggregated groups. 

Whether or not the state meets their state-determined performance level on a core indicator, the state is also required to identify and quantify gaps in performance on the core indicators between any subgroups or special populations 

of students and CTE concentrators overall, including “a quantifiable description of the progress each such subgroup or special population of students” has made in meeting determined performance levels. 

Perkins V also requires the state to publish actual levels of performance, including performance of special population and other disaggregated groups, publicly, in a variety of formats, including online. 

 

Local Performance Improvement Plan

The Perkins V legislation specifies that local recipients are also expected to meet at least 90 percent of their determined local levels of performance on the core performance indicators. If a local recipient falls below 90 percent of the determined local performance level, they will need to develop an improvement plan and implement it during the following year. 

Similar to the requirement for states, a local improvement plan must include an analysis of the gaps in performance among the special population and other disaggregated groups. 

Regardless of whether the eligible recipient meets 90 percent of the performance level for a core indicator, Perkins V requires disaggregate data to be reviewed annually for each indicator to identify and evaluate where there are disparities in performance between the subgroups or special populations of students and all CTE concentrators being included in the accountability measure.

CTE Accountability and Reporting Data–Secondary

In Minnesota, there are over 340 secondary districts of various types, including charter schools, that implement state-approved CTE programs. State-approved CTE programs are those which: 

• have been verified by MDE staff as being taught by an appropriately licensed CTE instructor (Minn. R. 3505.1000, Subp. 31A); 
• meet minimum standards for approval (Minn. R. 3505.1100); and, therefore 
• are eligible to receive Perkins funds (Minn. R. 3505.2400) and/or access the CTE levy (Minn. Stat. §124D.4531). 

Each district or charter school receiving Perkins funds must annually submit their P-File (Perkins) data to MDE (Minn. Stat. §124D.452; Minn. R. 3505.1800) via a secure web-based software application system. The person(s) reporting the P-file data are identified annually through the EDIAM system by a district’s superintendent. The secure, web-based software application system is open for data submission from mid-May to mid-September; official announcement and training dates are sent out upon the system’s opening. 

 

Secondary Data Collection Overview: Submitting a P-File (Perkins data) 

It is critical that a high degree of accuracy and integrity is maintained within the P-File data. Authorized users are asked to align their district’s P-File data with the programs and courses listed in the program approval database posted to MDE’s program approval webpage. 

After the P-File has been accepted into the system, it is important to review the Carl Perkins P-File Submission Status Report. This report is available through the Carl Perkins web application system (where the data are uploaded), or on MDE’s Carl Perkins Secure Reports webpage. This report offers a current- year summary of the data uploaded to MDE as well as a prior year comparison. District-authorized users are able to verify the accuracy of their P-File data by working through the information provided in the Help Document: P-File Checklist, posted on MDE’s data report webpage. The Help document will assist with verifying the data within the P-file Submission Status Report. 

MDE staff provide technical assistance. Staff at MDE collaborate with authorized users as well as administrators on an ongoing basis in order to provide resources and critical updates. 

Technical assistance is delivered through a variety of formats such as webinars, consortium-specific group trainings by request, conference presentations, and individual/personal communication. Contact MDE staff for scheduled webinar dates as well as any specific training or data needs. 

 

Secondary Data Collection Overview: Data Verification Process 

There are a number of ways in which the secondary data are systematically verified. 

Once a district submits their P-File data through the secure web- based software application, the software application performs an automated verification of each of the data elements within the P-File. For a list of all the data elements included in the P-File, please see the Carl Perkins Record Layout document listed within MDE’s data reporting website. 

The primary function of this automated process is to verify that: 

1. Each student Iidentification (SSID) number listed within the P-File also exists within the Minnesota Automated Reporting Student System (MARSS) database. 
2. Each school building listed in the P-File matches the district number it is reported under according to the information in Minnesota Department of Education-Organization Reference Glossary (MDE-ORG). https://education.mn.gov/MDE/about/SchOrg/ 
3. The program and course code combinations listed within the P-File align with the program and course code combinations listed in Table C (the Table C document lists all possible program and course code combinations, as well as teacher licensure information) found on the MDE Program Approval web page. 

A district’s P-File will not be accepted by the web-based application until it is deemed “error free” according to the standards set forth in the automated verification process. The data are stored in the Carl Perkins database which is maintained by MN.IT at MDE. 

After the P-File has been accepted by the web-based application, the district-authorized users are asked to review the P-file Submission Status Report on MDE’s secure reports website. Again, this summary report of the P-File data elements assists districts in identifying any reporting errors and verifying the accuracy of their data. 

It is ultimately up to the district authorized user to coordinate with CTE program folks in order to verify that all approved programs and associated courses which were implemented within a given school year have been included in the P-File submission. Consortium leaders, however, have access to this report for the specific districts within their consortium as well. 

Keep in mind that districts are able to submit updated P-Files as needed until the web-based system closes. The web-based application only retains a copy of the most recently submitted P-File. Staff at MDE also assist in monitoring the accuracy of the P-File submissions and contact district authorized users as needed. 

 

Secondary Data Sources 

It is important that the Perkins data have a high degree of accuracy and integrity because elements from the Perkins database are linked with four additional data sources and are summed across multiple years of data. 

All current year P-File data are verified against the SSID information in MARSS, specifically: student identification number, date of birth, and name. 

After all student level data have been verified, flags for participants and concentrators are created by summing the number of successful CTE course hours within one career field across a student’s entire high school experience until age 21 and prior to graduation. 

Finally, data from the Perkins database are linked with the following sources: 

1. MARSS: This system contains student demographic information and high school completion codes. 
2. Migrant Flag: This system is used to identify students who are documented migrant students. This information is used to link demographic data to assessment data and to report data to the U.S. Department of Education through the state’s CAR. It is not part of MARSS. It is stored in its own database. 
3. MCA/MTAS: These achievement test results are necessary for reporting academic achievement (1S1 and 1S2). 
4. SLEDS (Statewide Longitudinal Education Data System): This system contains information from both the Minnesota State system as well as from the National Clearinghouse (see Postsecondary Data sources for a description of the National Clearinghouse). This information allows us to fulfill the federal reporting requirements for 3S1 Post- Program Placement. 

 

Secondary Accountability: Reporting and Accessing Data 

All reports are created at the state, consortium, district, and individual school level. In order to access 

the Carl Perkins report on MDE’s secure reports webpage, superintendents are required to annually give permission to their staff through the Education Identity & Access Management (EDIAM) system. Consortium leaders who do not already have access should email the secondary data specialist to be given access. A number of data reports are currently available with more being added all the time. Secondary data reports fall into two broad categories: 

1. Accountability

• Performance Indicators: These reports align with the operational definitions listed at the beginning of this section (1S1, 2S1, 2S2, 3S1, 4S1, 5S3) and are organized according to the federal reporting structure. These are the reports to use when determining whether local levels of performance have been met and where gaps in performance might exist. 
• Enrollment: These reports display the number of CTE participants and concentrators, by career cluster. These reports are organized according to the federal reporting structure.