|
|
Pub Date: |
2003-04-00 |
Pub Type(s): |
Reports - Research; Speeches/Meeting Papers |
Peer Reviewed: |
|
|
|
|
Descriptors:
Accountability; Cutting Scores; Educational Research; Pass Fail Grading; Standard Setting (Scoring)
Abstract:
Standard-setting research has yielded a rich array of more than 50 standard-setting procedures, but practitioners are likely to be confused about which to use. By synthesizing the accumulated research on standard setting and progress monitoring, this study developed a three-dimensional taxonomy for conceptualizing and operationalizing the various procedures: outcome versus growth assessment, theory-driven versus data-driven approach, and observed scale versus latent scale mapping.An empirical study is reported to illustrate how these various approaches can be implemented to meet the accountability challence in the No Child Left Behind era. Consistency analysis of 12 standard-setting procedures reveals vastly disparate pass/fail decisions among different procedures, even within the same conceptual framework or mapping operation. Particularly disturbing is the finding that the passing rate may jump from as low as 29% to as high as 79%, depending on whether the standard is mapped to the observed-score scale or the latent-score scale. Implications and future directions for policy makers, school officials, and psychometricians are discussed. (Contains 9 tables and 107 references.) (Author/SLD)
Note:The following two links
are not-applicable for text-based browsers or screen-reading software.
 Show
 Hide
Full Abstract
Related Items: Show Related Items
Full-Text Availability Options:
 ERIC
Full Text (794K)
|
|
|
Pub Date: |
2002-00-00 |
Pub Type(s): |
Information Analyses; Opinion Papers |
Peer Reviewed: |
|
|
|
|
Descriptors:
Adult Education; Agency Role; Competency Based Education; Compliance (Legal); Credentials; Education Work Relationship; Educational Certificates; Employment Qualifications; Evaluation Criteria; Licensing Examinations (Professions); Occupational Tests; Personnel Selection; Postsecondary Education; Professional Associations; Secondary Education; Selection Tools; Standard Setting; Standards; Student Certification; Vocational Education
Abstract:
Career-technical educators face three issues in credentialing through assessment. First, the occupational credentialing domain is large and evolving, and a clear understanding of it is a prerequisite to considering adoption of a credential. Three types of credentialing are registration, certification, and licensure. Credentialing organizations are categorized by their mission (government regulatory board, trade association, vendor-specific, National Skills Standards Board). Oversight organizations are professional organizations that disseminate information and provide voluntary oversight by evaluating credentialing systems. Second, a set of clear, comprehensive standards is needed to define credential quality and credibility. Proposed evaluative criteria/standards to select assessment-credentialing are marketability, recognition, alignment to curriculum, quality of input standards, quality of assessments, and usability for career-technical education (CTE) setting. Third, CTE policymakers and educators need a rational, efficient process to evaluate credential systems and associated assessments against a set of standards. The following nine steps are the process: (1) define purposes and uses of occupational credentialing systems; (2) set evaluation criteria; (3) identify credentialing systems and evaluate preliminary link to programs; (4) conduct initial screening; (5) determine quality of input standards; (6) determine quality of credentialing assessments; (7) conduct final linkage to curriculum; (8) determine marketability and recognition; and (9) develop data collection procedures. (Contains a 53-item bibliography.) (YLB)
Note:The following two links
are not-applicable for text-based browsers or screen-reading software.
Show
Hide
Full Abstract
Related Items: Show Related Items
Full-Text Availability Options:
ERIC
Full Text (469K)
|
|
|
Pub Date: |
2002-00-00 |
Pub Type(s): |
Information Analyses |
Peer Reviewed: |
|
|
|
|
Descriptors:
Accountability; Adult Education; Educational Research; High Stakes Tests; Postsecondary Education; Secondary Education; Standardized Tests; State Programs; Student Evaluation; Test Interpretation; Test Use; Test Validity; Testing Programs; Vocational Education
Abstract:
The topic of high-stakes testing (HST) is important because HST has direct and indirect effects on career-technical education (CTE) programs and timely because HST increasingly enters public discussion and has produced a large body of research and practice that generalizes to CTE. A review of HST has identified two persisting dilemmas: policy and public expectations of testing exceeding tests' technical capacities and tension between testing to increase fairness and testing to classify. Applicable strategies to provide validity for HST are reliability estimation of scores used to make decisions, expert judgements of item linkage to curricula, studies of the predictive power of HST scores, and studies of consequences. Two opposing perspectives on the accountability-testing theme are that use of HST for accountability is a positive application of data-driven management to education and that the consequences of HST are negative. Descriptions of HST systems in Kentucky, Texas, and Massachusetts indicate different ways to accomplish HST; use of advisory panels to represent stakeholders' viewpoints; and continuous change. Findings of an e-mail survey of state CTE directors suggest ways to expand assessment modalities--computer delivery of assessments and authentic assessments or multimodal assessments that include high- and low-stakes components. Implications are that the CTE community needs awareness of HST; tests should be used responsibly; and a useful database system should be developed. (Contains 76 references and a list of 8 Internet sites.) (YLB)
Note:The following two links
are not-applicable for text-based browsers or screen-reading software.
Show
Hide
Full Abstract
Related Items: Show Related Items
Full-Text Availability Options:
ERIC
Full Text (260K)
|
|
|
Pub Date: |
2000-11-00 |
Pub Type(s): |
Information Analyses |
Peer Reviewed: |
|
|
|
|
Descriptors:
Career Education; Computer Assisted Testing; Computer Oriented Programs; Computer Uses in Education; Educational Assessment; Educational Technology; Evaluation Methods; Item Analysis; Item Response Theory; Performance Based Assessment; Postsecondary Education; Secondary Education; Student Evaluation; Technical Education; Technological Advancement; Test Format; Vocational Education
Abstract:
The process of assessment in career and technical education (CTE) is changing significantly under the influence of forces such as emphasis on assessment for individual and program accountability; emphasis on the investigation of consequences of assessment; emergence of item response theory, which supports computer adaptive testing; and pressure for authentic assessment by critics of standardized testing. The use of technology for assessment is being increasingly stressed. Potential advantages of computerized assessment include rapid feedback, money savings, enhanced security, more curriculum time, and capability to track process-oriented variables. Potential disadvantages include costs of equipment, personnel, and training; increased marginalization of groups based on ethnic or socioeconomic status; and missed opportunities to implement authentic assessments. Examples of applications of technology to evaluation in CTE include test design, item creation, presentation, item scoring, and location (through use of the Internet). In moving toward computerized assessment, consequences should be considered in line with current evaluation and validation models. Especially for policy makers and administrators, implementation is an important aspect of computerized assessment. Technology can support the broad changes in assessment that are ongoing as a function of internal and external scrutiny, and both authentic as well as traditional assessment can benefit from technology. (Contains 24 references.) (KC)
Note:The following two links
are not-applicable for text-based browsers or screen-reading software.
Show
Hide
Full Abstract
Related Items: Show Related Items
Full-Text Availability Options:
ERIC
Full Text (145K)
|
|
|
Pub Date: |
1999-00-00 |
Pub Type(s): |
ERIC Publications; ERIC Digests in Full Text |
Peer Reviewed: |
|
|
|
|
Descriptors:
Adult Education; Career Counseling; Career Education; Cultural Awareness; Cultural Differences; Cultural Pluralism; Cultural Relevance; Culturally Relevant Education; Educational Trends; Evaluation Methods; Secondary Education; Test Bias; Test Reliability; Test Validity; Trend Analysis; Vocational Evaluation
Abstract:
Multicultural perspectives on assessment challenge traditional perspectives by advancing an additional source of variation in test responses that is presumed to escape test developers and test users. Increasing and convergent evidence from multiple sources indicates the following types of ethnocentric errors in test development, administration, and interpretation: misdiagnosis, labeling, inappropriate treatment planning, and erroneous conclusions about intervention effectiveness. In 1995, Sedlacek and Kim described four common errors in career assessment. Focus on multicultural assessment issues has since increased; however, the main focus has been on assessments of abilities and psychopathology. Career assessment has traditionally concentrated on assessment of the individual and has emphasized measurement of abilities, personality traits, interests, and various career theory constructs (maturity, indecision, self-efficacy). Reviews should be conducted of the major instruments available for classifying individuals in terms of abilities, personality traits, and vocational interests to determine their fairness to individuals from different cultural backgrounds. Attention to multicultural assessment in the area of typical performance is still in its "infancy" relative to the area of maximal performance, which is in its "adolescence." Test developers could profit from considering and implementing the concept of Sensitivity Review Guidelines and Principles, which are available on the World Wide Web. (Contains 28 references.) (MN)
Note:The following two links
are not-applicable for text-based browsers or screen-reading software.
Show
Hide
Full Abstract
Related Items: Show Related Items
Full-Text Availability Options:
ERIC
Full Text (30K)
|
|
