Cris J. Costa
Irene Shipkova
LMU/PSY542
Dr. Mills
SUMMARY
Using the Internet for Psychological Research:
As the popularity and use of the Internet has increased, psychologists have become very aware of the potential uses of the Internet for research. As a result, numerous researchers conducted studies to examine the validity of Internet-based psychological research. Buchanan & Smith, the authors of this article, analyzed the work of various researchers, specifically the methodology of the research techniques used; the threats to reliability and validity of Internet-based results; and the ways in which mediated test differs from traditional paper-and-pencil testing.
According to Buchanan & Smith, psychologists quickly realized the initial benefits of using the Internet and computerized testing for research. It allowed them to access a larger amount of potential research participants; minimize costs of testing; automate test administration; automate data acquisition, scoring, and analysis; and easily locate and contact people with special characteristics needed for the project. At the present time, the majority of Internet-based studies are more questionnaire-based, with personality tests and surveys being the most popular. However, there is research that suggest that computerized versions of the traditional personality tests are generally equivalent to their paper-and-pencil various, given that the test are (a) not speeded and (b) require some form of multiple or forced choice response to textual items (Buchanan & Smith, 1999). Following is a summary of the Self-Monitoring Scale (SMS-R) that was conducted to see whether or not Internet-based tests were equivalent to traditional test formats.
For this study, two groups were recruited. The first group (sample 1) was recruited via Usenet newsgroups in which messages recruiting participants for on-line test had been previously seen. The messages for this particular study were posted four times at two-week intervals for two months. A total of 1181 participants were recruited. 963 participants actually took the test using the Internet-based questionnaire. This group was comprised of 491 males, 472 females; 45 were students of some type and the age range was between 11-67 years old
(M = 32 years, SD = 10.79). Most of the respondents were from the
The comparison group (sample 2) was recruited through traditional means. All of the members were undergraduate students of a local university. This group had 224 undergraduate volunteers comprised of 35 males, 176 females, and 13, who did not state gender. The age range was between 28-53 years old (M = 27, SD = 8.10). All the participants were tested in small groups after classes using the traditional paper-and-pencil version of the SMS-R.
After extensive analysis, the following information was found (see Table A for more detail):
· The first group indicated some level of fit that was better than the comparison group;
· The characteristics of the two groups seemed fairly similar and an independent samples t test indicated no significant different in means;
· The Internet-based test seemed to have better psychometric properties than the traditional version of the test;
· Reliability for the Internet-based test was slightly higher than the traditional version;
· The first factor and the total score suggest that the Internet may actually provide better measure;
· Increase in honesty and self-revelation with members from sample 1; and
· Greater heterogeneity amongst the participants from sample 1.
These findings may indicate that an Internet-based test may be a better measure of a personality trait than the traditional version only because of the increaser in honesty, self revelation, greater homogeneity with sample a, however, the findings do present the potential benefits of the Internet for research. Even though they were not able to detect any differences between the factor structures of the samples does not mean that the Internet-based version is the best measure of self-monitoring. The bigger issue is the validity of Internet-based testing as a measure of self-monitoring. This study only revealed to us the reliability of the Internet-based test, not about it construct validity.
Moreover, additional questions remain to be answered:
· What is the subjective experience of a participant in the study?
· Why would a person participate in such study?
· Of the people who saw the recruitment message, how many actually visit the test’s Web page?
· Of the people who visit the Web page, how many complete it and submit then answers for analysis?
· What sets them apart from those who do not?
In order to properly investigate the reliability of the power of self-monitoring, we must examine the sampling strategy. The Internet sample (sample 1) was recruited primarily from psychology oriented newsgroups. Studies indicate that men are the majority users of the Internet. Another sampling problem is that comparison group was drawn from a pool of people comprised of predominately psychology students. The psychology field, which is dominated by women, may cause a misrepresentation of a general sample. The fact that the groups sampled would have a lot in common, may skew test results.
It is evident that although these additional tests prove positive for Internet-based testing, additional work is required that must compare the scores of groups of Internet-recruited participants vs. traditional paper-and-pencil tests.
Table A
Fit Sample 1 Sample 2
Chi 669.70 229.57
d.f. 132 132
GFI 0.925 0.900
AGFI 0.903 0.871
RMS 0.014 0.015
NFI 0.756 0.670
TLI 0.759 0.791
References
Buchanan, T., & Smith, J. (1999) Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, Feb 99, Vol. 90, Issue 1, 125, 20. Retrieved
Using the Internet for Psychological Research:
a. Lab space
b. Testing time
c. Materials
B. THREATS TO RELIABILITY AND VALIDITY OF WEB-BASED TESTS
a. Heterogeneity of Internet users is increasing
b. Population of Internet users believed to be predominately male
c. Women are strongly represented in newsgroups and tend to be more interested in psychology and personality testing
a. Participants have different motivations
3. Environmental factors and nature of the testing environment
a. Computerized testing have more control over testing conditions and
consequently greater objectivity
a. Various softwares, configurations, different displays
b. Slow connections can lead to frustration
a. Participants have control over their submissions using their browswer
C. SELF-MONITORING SCALE
a. other-directedness
b. extraversion
c. acting ability
a. Sample 1-tested using Internet questionnaire
b. Sample 2-undergraduate student volunteers
D. RESULTS
1. The fit indices obtained using Sample 1 are better than the comparison group
2. Means and distributions of scores
a. Characteristics of the samples appear to be similar
b. Independent samples t test indicated no significant different in means
E. DISCUSSION
1. Possibility that the Web-based test has better psychometric properties than the
traditional version
a. Reliability slightly higher
b. Higher correlation between first factor and total scale score
c. Increased honesty and self-revelation
d. Greater heterogeneity of Internet sample
2. Validity is questionable
a. Internet participants were mainly from psychologically related newsgroups and
traditional sample consisted of psychology students (not much variance)
F. CONCLUSION
1. Great potential using the internet for psychological research
2. Can an internet test be considered equivalent to the traditional version?
a. Must look at each test individually
b. See each use to which it is put
3. SMS-R proved its equivalency