Tag Archive for: Test Review(Gordon Diagnostic System) homework help

Tests and Measurements

Achievement Test Review

 

Review of the Becker Work Adjustment Profile 2

Reviewer 1: James T. Austin & Stephanie D. Tischendorf

Reviewer 2: Pam Lindsey

 

Description: The Becker Work Adjustment Profile 2 was created to measure the vocational

competency of people with disabilities in their work environments. By assessing work habits, attitudes, and skills of people with special needs, it aims to target problem areas and assess the level of

supports needed. It is targeted for those ages 13 and older who are disabled, including those who

are mentally retarded, physically disabled, emotionally disturbed, learning disabled, and-or

economically disadvantaged. The instrument is completed by a rater-observer who, according to

the user’s manual and test booklet, have “closely observed the daily work behavior of the client and

has knowledge of the individual’s work adjustment”. Austin & Tischendorf describe the Becker

Work Adjustment Profile 2(BWAP:2) as a “restandardization of the 1989 Work adjustment

profile (as the items have not changed).” They also offer information about the test’s first version

from 1989, which was a revision of a rating scale developed as part of a 1965 Federal grant.

Reviewer 2, Pam Lindsey, points out that vocational competence is an element of rehabilitation

in those with disabilities, thus this test is useful for those working with disabilities and helping

them to be successful in the workplace. By knowing a clients vocational competence, we can

help address areas the client may be having problems in that affect their job performance.

 

Types of Items:

 

The first page of the instrument is used to gather information on the patients background

(name, sex, date, grade, date of birth, age in years, IQ, school-facility, primary disability,

secondary disability, name, & title of evaluator). Pages 2 through 12 include 63 ratings that are

divided into four categories. Work habits and attitudes is the first category, with 10 items or

ratings questions, Interpersonal relations is the second with 12 items, Cognitive skills is the third

with 19 items, and work performance skills is the last with 22 items. Page 13 and 14 provide

information on the results of the test and psychometric information. The rater scores each

domain and enters the score at the end of each domain. Vocational personnel familiar with daily

demands of the job and the individual being assessed complete the questionnaire (Pam Lindsey).

 

Scoring Information:

The items are rated on a 0-4 rating, with 0 being a negative score and 4 being a positive

score. Four domains and a total composite score (Broad Work Adjustment) are used in creating

a profile of the client and work placement that can be linked to work support needs. As stated

above, the rater scores each domain and enters the score at the end of each domain. Raw scores

are translated to T scores and percentile ranks by disability category. Again, this information can

be found on page 13 of the questionnaire booklet. Also found on page 13 is raw scores,

percentiles, work placement and work support needs, and other useful information for

interpreting the results of the questionnaire.

 

 

 

Technical Aspects/Psychometric Properties: A user’s manual is included with the exam that

Austin & Tischendorf feel is quite “extensive”. The user’s manual does 2 things. It defines the

four major methods of client work evaluation (work sample, job analysis, standardized tests and

situational assessment), this test falling into the situational assessment category. It also discusses

vocational competence and how it relates to work adjustment. It provides information on

administering the test, scoring the test, and use of the test. It also provides technical evidence of

reliability and validity.

The BWAP:2 was normed against 4,019 individuals with various disabilities, although Austin &

Tischendorf felt it was weighted towards those with MR. The norm group is categorized by

diagnostic category (disability) and gender, and the sample is geographically diverse. All

normative data is found in Appendix A of the user’s manual.

Reliability: Different estimates of reliability of the BWAP:2 is shown, derived from subsamples

based on diagnostic category. Internal Consistency estimates for domain (.80 to .93) and for

BWA total score (.87 to .91), retest estimates of over 2-week interval (range .82 to .96 across

domains), standard errors of measurement (.91 to 5.84) and interrater reliability estimated by

pairs of raters for a sample of 117 adults in 3 sheltered workshops (.82 to .89 across domains, .87

for Broad Work Adjustment). Both Austin & Tischendorf and Pam Lindsey felt that these

numbers signified multiple estimates of reliability and normally accepted values.

Validity: The user’s manual also provides information on validity. Construct and criterion-

related validity data were stated to be sufficient. To measure criterion-related validity, 167

people with MR had their scores on the BWAP:2 compared with their scores on the AAMR

Adaptive Behavior scale. The AAMR Adaptive Behavior Scale measures vocational adjustment

and adaptive behavior in its test takers. The evidence from this comparison was used as proof

that the criterion-related validity was satisfactory. Internal consistency, test-retest and interrater

reliability studies were conducted and all showed to be adequate and/or stable. Pam Lindsey felt

that “Overall, technical data appear to support adequate reliability and validity”. Austin &

Tischendorf, however, felt that really the task of comparing the AAMR Adaptive Behavior Scale

and the BWAP:2 was showing convergent validity, as they felt this did not show how the

instrument predicts success in some external outcome.

 

Possible Accommodations and modifications: Austin & Tischendorf felt the test includes an

economically disadvantaged subgroup that they feel needs to be better supported to be beneficial.

Another modification idea by Austin & Tischendorf was to increase the length of the retest

period to more than 2 weeks in an attempt to strengthen the reliability evidence of the test. They

also felt that the breakdown of the standardization sample or the norm group was not broken

down enough. They wrote that it would be better if it were additionally broken down into age,

ethnicity, and sex. Currently, it is broken down into mean age by disadvantage. They also

thought statistically significant differences in regard to these break downs (age/ethnicity/sex)

should also be included.

 

Practical Applications and Uses: As stated in the description, the BWAP:2 was made to assess

work habits, attitudes, and skills of those with disabilities. By doing this, we can assess the level

of support needed for these people with disabilities and help them get the support that they need

to perform a job the best they can. It should be used as part of a larger assessment process and

not solely used to assess one’s capabilities of performing or holding a job. Instead, it should be

used to help find weak areas that need help or services.

 

 

 

 

Clinical recommendations and cautions for use: Austin & Tischendorf caution that the

instrument is emphasizes mental retardation and may be dated when compared with current

theories and approaches to learning disabilities. They also recommend using modern test theory

and confirmatory factor analysis in analyzing data and in appraising the construct validity of the

four different domains and the BWA composite score. They also felt that it is important that the

vocational competence is standardized across all raters in order to establish the validity of

observational data.

 

Summary of Reviewers:

 

In general, Pam Lindsey provides a less extensive review of the BWAP: 2 than do James T.

Austin & Stephanie D. Tischendorf. Pam Lindsey seems to give the basics needed to get a

picture of the BWAP:2. Pam Lindsey mentions the bias that comes along with having an

observer who is observing the client or test taker and that perceptions and prejudices can get in

the way. However, she feels that the instrument is valuable in helping professionals try to

measure the vocational competence of persons with disabilities, targeting areas that need special

attention, and assist them in building a rehabilitation plan that will be appropriate for their

special needs. She feels that it is technically adequate or sound and that scores could be

compared with other measures like adaptive behavior or cognitive abilities. She also feels it is

not meant to be an assessment of an individual’s ability to work or be successful on the job but

rather to be part of a bigger plan or process to help the professional target areas that are strengths

and weaknesses to help them get support that they need.

 

Austin & Tischendorf offer a much more in depth and critical review of the BWAP:2. While

Pam Lindsey felt that overall the data showed satisfactory reliability and validity, Austin &

Tischendorf felt that comparing the AAMR Adaptive Behavior Scale and the BWAP:2 did not

prove that the BWAP:2 could predict success in some external outcome, which is basically the

definition of criterion related validity. Pam Lindsey also did not offer many ideas in terms of

modifications or problems with the test. Austin& Tischendorf felt the norm group was heavily

weighted towards those with MR, and the test is for those with who are mentally retarded,

physically disabled, emotionally disturbed, learning disabled, and-or economically

disadvantaged, and thus should have a norm group more equally weighed between the different

disabilities. They felt that including an economically disadvantaged should be better supported

to be beneficial, and that the test-retest period should be made longer and should be compared

with week 2 re-test results in order to strengthen the reliability evidence of the test. They also

called for a more extensive breakdown of scores(age/ethnicity/sex) should be incorporated into

the BWAP:2. In terms of final thoughts about the BWAP:2, Austin & Tischendorf felt the test

was easily administered and scored by raters, with “ample opportunity to observe the focal

individual” (Austin & Tischendorf). They felt that several improvements had been made from

its previous version. They left it by stating the following:

“Any issues that remain do not preclude a recommendation to use the BWAP:2 but rather

suggest continuing to develop its knowledge base”(Austin & Tischendorf).

 

 

 

 

 

My Opinion of the Instrument/Reviewers: Overall, I felt that the Becker Work Adjustment

Profile 2 seems it could be a very useful tool for helping those with disabilities be the best that

they can be. I agreed with both about the reliability being adequate, however I sided with Austin

& Tischendorf when they discussed the issue of proving evidence that the BWAP:2 has criterion

related validity. By comparing tests, I do not see how that proves that BWAP:2 proves a client

would be successful in the future. I also agreed with Pam Lindsey about the bias that can come

along with having an observer that is rating someone else, and the interrater reliability that goes

along with this. However, I also agree with Pam Lindsey that this comes with the territory of all

observational instruments, and I feel it should be kept in mind when interpreting scores. Also,

the BWAP:2 does have adequate interrater reliability at .82 to .89 across domains, and that I

think should put most those considering using the BWAP:2 at ease.

 

I also feel that in terms of reliability and validity that it is sufficient and is an advantage to using

this test. I think that the goals of this test also really make it special or worth using, because it is

looking for strengths and weaknesses in the client’s vocational skills, habits and attitudes. As a

rehabilitation counselor this is a huge goal. Having a way to try to find these weaknesses and

helping a client work on them or receive help can help a client’s job stability, self esteem, and

help them feel more socially accepted by others because they are performing up to standards.

 

As Pam Lindsey mentioned, I think it is very important with all tests to remember that it should

be used as part of a larger process which includes talking to and learning about the client and

their experiences/what they might already know about themselves, other tests or inventories, and

information about the disability. Tests should never be thought of as a sort of as an

unquestionable source of information. It is also important to look at how appropriate the test

might be for the client. From what I have read about the test, it may not be the best test for

someone with learning disabilities due to seeming a little dated in terms of learning disabilities,

however depending on the case maybe there is information to gather that could be useful by

using the BWAP:2. Every situation is different and it is part of the rehabilitation counselor’s job

to figure out if a test is appropriate for the user. References:

Information from the reviews of James T. Austin & Stephanie D. Tischendorf , and Pam Lindsey

The Becker Work Adjustment Profile 2

http://ovidsp.tx.ovid.com.gate.lib.buffalo.edu/spa/ovidweb.cgi

 

1)The Test- cost, time to take the test, theory behind the test, number of items, age appropriateness, and any other information relevant to teaching me about the test ( Approximately one page double spaced)

2)Reviewer #1- norm sample, practicality and cultural fairness, validity, reliability, final comments  ( At a Minimum, one page double spaced)

3)Reviewer #2- norm sample, practicality and cultural fairness, validity, reliability, final comments ( At a Minimum, one page double spaced)

4) Your thoughts on norm sample, practicality and cultural fairness validity, reliability, final comments about using the test. Why or why not. (At a Minimum, one page double spaced).  I want your thoughts based on specific information and not just opinions such as “I don’t like the GRE’s” or “I don’t think it’s fair to subject students to standardize testing.”  I want to know what you think about the norm sample, practicality and cultural fairness validity, reliability based   specifically on what you learned from both reviewers and any other source.