Before you can study an effect, evaluate customer's attitudes toward a new product, or assess whether a new tutoring program improved student grades, you have to be able to measure and assess the outcomes. Let our statistical consultants help you develop an instrument, evaluate its reliability and validity, and analyse your results. While there are many analytic tools available in psychometrics, here are a few examples of the methods we use to meet our clients' needs.
If you do not have an instrument yet, we can help you write questions, and use Exploratory Factor Analysis (EFA) to identify how many unique constructs are being measured and which questions are the most informative to assess your outcomes. We know that items are not always continuous. Our statistical consultants are expert in both traditional continuous exploratory factor analysis and modern item response theory (IRT), so we can conduct exploratory analyses on continuous data, categorical data including binary, ordinal, or nominal, or any combination of the two.
Although exploratory factor analysis is a useful tool to find patterns, exploratory analysis can capitalize on chance and is less likely to replicate in a new sample. Once a construct and factors are identified, Confirmatory Factor Analysis (CFA) can be used to verify that the factor structure found in the exploratory analysis fits the data in a new or hold out sample.
Sometimes, samples are very expensive to collect and it may not be feasible to have a large enough sample to split in two for exploratory and confirmatory analyses. Our statistical consultants can employ K-fold cross-validation techniques from the machine learning and data mining field to psychometrics in order to reduce the effects of capitalizing on chance and exploratory analysis.
Once an instrument is created to measure a construct, we will evaluate the construct validity using multiple techniques. For example, to demonstrate convergent validity, a scale measuring attitudes about Company X, should be associated with customers behavior of buying Company X's products. Conversely, discriminant validity is assessed by ensuring that attitudes about Company X are not associated with attitudes towards other, unrelated companies.
Having a valid instrument is a good first step, but it is also important to have a reliable instrument. Our statistical consultants can calculate measures or internal consistency reliability, such as the traditional Cronbach's alpha coefficient, as well as recent ones such as coefficient omega, along with confidence intervals so you know how accurate that internal reliabiltiy estimate really is.
We can also calculate test-retest reliability or the intraclass correlation coefficient to assess how stable the meausrement is over time and repeated measures.
Established instruments may not function the same way in different populations. For example, children and adults may interpret identical questions differently, leading to the same instrument measuring different constructs. Utilizing techniques such as multiple group confirmatory factor analysis or interactions, we can help you assess measurment invariance across different populations or groups or whether some items show differential item functioning.
We can also use multiple group CFAs to establish that an instrument is functioning the same way over time as a first step to assessing change over time. For example, if an instrument is assessed before and after an intervention, it is important to establish that the instrument is functioning the same in both the intervention and control groups, before and after intervention, before concluding that a chance in scale values is due to a change in the construct of interest and not in what is being assessed or measured.
Premier: Our most bespoke offering. Let our consultants create solutions and reports that turn your data into strategic insight.
Academic: For accredited, not-for-profit universities. From methods write up assistance to advanced model analysis, we are experienced researchers who enjoy scientific discovery.
© 2013 - 2017 by Elkhart Group LLC