Construct validity refers to whether an assessment measures a theorized psychological construct. In the case of SmarterMeasure, construct validity is a measurement of the degree to which SmarterMeasure is an indicator of a learner's level of readiness for studying in an online or technology rich environment. Results from the three studies described below indicate that SmarterMeasure has strong construct validity in that it is an indicator of the goodness of fit for distance learning as is evidenced by multiple correlations that are statistically significant at the .01 level.
It should be noted that SmarterMeasure is not designed to be a predictor of academic success. There are a myriad of variables which impact academic success in online courses ranging from the student's intelligence to the level of interactivity of the online faculty member. SmarterMeasure is an indicator of the degree to which online, hybrid and technology rich courses are a good fit for a student. SmarterMeasure does not make a value judgment indicating that a student should or should not take the courses. Rather it informs the student of their strengths and opportunities for growth in areas related to taking these type courses. If a student is indicated to be deficient in a certain area and then if the school provides appropriate remediation and/or support, then SmarterMeasure can serve as a retention tool by helping students succeed as they learn in the context of online or technology rich courses.
In 2011 a major for-profit university conducted an extensive validity study to determine if SmarterMeasure was being an accurate indicator of the student success variables of academic achievement, engagement, satisfaction and retention. Statistically significant relationships were found between SmarterMeasure scores and each of these four constructs. A summary of these findings is provided below you can read a copy of the final report of Phase One and Phase Two of this study.
Academic Achievement and Retention were compared to SmarterMeasure scores using grade and enrollment data.
- The measures of Individual Attributes, Technical Knowledge, and Life Factors had statistically significant mean differences with the measures of GPA.
- The measure of Learning Styles had a statistically significant mean difference between students who were retained and those who left. A 73% classification accuracy of this retention measure was achieved.
- The measures of Individual Attributes and Technical Knowledge were statistically significant predictors of retention as measured by the number of courses taken per term.
Satisfaction and Engagement were compared to SmarterMeasure scores using students' responses to an online survey.
- The measures of Individual Attributes and Life Factors had statistically significant mean differences on six of the seven survey items. Reading Rate, Technical Knowledge, and Technical Competency had significant differences on four of the seven items.
- The measures of Individual Attributes and Technical Competency had statistically significant relationships with the four survey items related to Engagement. The items of hours per week spent on course related activities; number of times per week logging into course; length of discussion board postings; and number of times contacting technical support can be predicted given knowledge of Individual Attributes, and more specifically the subscales listed.
- The measures of Life Factors, Individual Attributes, Technical Competency, Technical Knowledge, and Learning Styles were used to correctly classify responses to the survey questions related to engagement and satisfaction with up to 93% classification accuracy.
- Structural equation modeling was used to create a hypothesized theoretical model to determine if SmarterMeasure scores would predict satisfaction as measured by the survey. Results indicated that prior to taking online courses, student responses to the readiness variables were important indicators of later student satisfaction/retention. The structural coefficient for Ready predicting Satisfy, = .36, was statistically significant (z = 6.01, p = .0001). Therefore, the multiple SmarterMeasure assessment scores are a statistically significant positive predictor of the survey responses.
Further analysis revealed that the predictive nature of SmarterMeasure scores as classified by the Readiness Ranges can be improved using recommended adjustments to the grading thresholds.
The majority of survey participants (90%) either somewhat or definitely remembered taking the assessment. The majority of survey participants (89%) found the assessment somewhat useful, useful, or very useful; while only 11% did not find it useful at all as a student service.
Phase two of the study drilled down into the data at the sub-scale level Statistically significant relationship were found between SmarterMeasure data and student success categories related to academic success and retention. The table below indicates which sub-scales had statistically significant relationships with these key performance indicators.
|SmarterMeasure Scale||Readiness Domain Subscales|
Postive vs. Negitive
Pass Vs. Fail
|Life Factors||Place, Reason, and Skills||Place|
|Learning Styles||Social and Logical||N/A|
|Personal Attributes||Academic, Help Seeking, Procrastination, Time Management, and Locus of Control||Time Management|
|Technical Competency||Internet Competency||Internet Competency and Computer Competency|
|Technical Knowledge||Technology Usage and Technical Vocabulary||Technical Vocabulary|
A predictive model using multiple regression was created to measure the degree to which SmarterMeasure sub-scales are predictors of academic success as measured by GPA. Each set of subscales for the Readiness Domains were considered a theoretical set of independent predictor variables, therefore separate regression analyses were conducted on each. The table below illustrates that GPA was significantly predicted by Place, Skills, Verbal, Logical, Help Seeking, Time Management, Locus of Control, Computer Competency, Internet Competency, and Technology Vocabulary.
|Life Factors||Place and Skills||12.35||.0001|
|Learning Styles||Verbal and Logical||3.95||.02|
|Personal Attributes||Help Seeking, Time Management, and Locus of Control||22.11||.0001|
|Technical Competency||Computer and Internet Competency||22.75||.0001|
|Technical Knowledge||Technology Vocabulary||38.76||.0001|
In 2007 an external research firm (Atanda Research, Alexandria, VA) was commissioned to analyze the data gathered during a study concerning the relationship of SmarterMeasure scores and measures of academic success and goodness of fit of distance education as a measure of construct validity. The major findings of this report were that there were forty-two statistically significant correlations between SmarterMeasure variables and measures of academic success and goodness of fit. Of the five constructs measured by SmarterMeasure, the construct with the most correlation to academic success and goodness of fit was Individual Attributes. The variable of the participant's individual attributes scores were statistically significant at the .001 level with all measures of academic success and goodness of fit. The variable with the strongest correlation in the study was relationship between Grade Point Average and Reading Comprehension. Click here to view a copy of this report.
In 2008 the study conducted by Atanda Research was replicated as a part of a learner's dissertation research which involved 2,622 students who had taken SmarterMeasure representing over 300 schools. This replication yielded even stronger results than the original study. Of the possible 105 correlations measured, 74 were found to be statistically significant. The factor measured by SmarterMeasure that had the strongest correlations to measures of goodness of fit and academic success was individual attributes which yielded correlations in each of the seven categories which were statistically significant at the .01 level. This finding mirrored the finding from the 2007 study which also indicated that individual attributes were the strongest indicator of goodness of fit of distance education.
The following correlation matrix presents the results of the statistical analysis from this study:
|SmarterMeasure Scores||Measures of Goodness of Fit||Measure of Academic Success|
|Reading Required||Find Time||Computer Skills||Internet Access||Good Choice||Take Another||GPA|
|Overall Tech Competency||.013||-.014||.170**||.154**||.114**||.109**||.144**|
|Visual Learning Style||0||-.007||.041*||.008||.013||-.012||.014|
|Social Learning Style||.082**||.061**||.095**||.067**||.047*||.039||.003|
|Physical Learning Style||-.007||.005||-.003||.001||-.004||-.016||-.038|
|Aural Learning Style||.037||.04||.103**||.081**||.033||.022||-.011|
|Verbal Learning Style||.162**||.101**||.143**||.119**||.131**||.102**||.073**|
|Solitary Learning Style||.091**||.072**||.089**||.076**||.085**||.074**||.067**|
|Logical Learning Style||.115**||.079**||.157**||.144**||.126**||.108**||.071**|
* Correlation is significant at the .05 level
** Correlation is significant at the 0.01 level