The Benefits of Quality Matters Certification:
What the Analytics Reveal

a report produced by FIU Online
July 2016
(download PDF version of report)

Background and Intent

FIU Online engaged in research to highlight the benefits to both students and faculty when taking and teaching an online course that has received the Quality Matters certification. FIU Online leveraged the data available via the Blackboard Lean learning management system to gain this understanding.

With this research, we sought to understand if there were differences in the student learning experience with courses that had received QM certification versus those that had not received QM certification.

The courses used in this research were fully online courses taught at Florida International University in the Spring 2016 semester. Florida International University (FIU) is an American metropolitan public research university located in Miami, Florida, United States. FIU Online is the centralized distance-learning department for Florida International University. Quality MattersTM (QM) is a faculty-centered, peer review process that is designed to certify the quality of online courses and online components.

As of the date of this report, FIU has the second highest number of QM certified courses in the nation.

Methodology

All data was pulled from the Blackboard Learn learning management system (LMS). The courses used in this research were fully online courses taught at Florida International University in the Spring 2016 semester.

Our sample included all fully online courses taught in Spring 2016, separated by courses ‘Quality Matters’ certified and not ‘Quality Matters’ certified. This report only includes active sections in the semester and those marked as mode “I” (Internet/Fully Online). For comparison where student data is aggregated, this report filters only students that kept their enrollment in the course.

We reviewed the following metrics to determine if QM courses had better, same or lower percentages as it related to:

  • interactions
  • item count (assessment)
  • item count (content)
  • item count (tool)
  • submissions
  • average course accesses
  • average minutes spent in course
  • course evaluations

For specific definitions of the items above, please review the Appendix section of this report.

Insight #1: QM Courses had 16% more student interactions than non-QM courses

On average, students taking a QM course were interacting with the course 16% more than students enrolled in a course that had not been QM certified. An interaction is defined as a user clicking into an item in Blackboard. Many types of student actions can lead to an interaction being logged, such as accessing a course item or submitting a discussion forum post. QM Standard #5 is “Learning Activities and Learning Interaction”. This standard ensures that course activities promote learning objectives set for the course.

Insight #2: QM Courses had 19% higher rate of student submissions than non-QM courses

More work, on average, is being submitted by students enrolled in QM courses versus students in enrolled in courses that are not QM certified. A submission is defined as a student submitting an assignment, blog, discussion, journal, Safe Assignment, survey, or test submissions. One of the QM Standards is “Assessment and Measurement”. The QM rubric checks for courses to have assessments that are “sequenced, varied and suited to the learner work being assessed”.

Insight #3: QM Courses had 10% higher course accesses than non-QM courses

This refers to a student clicking into a course from the Blackboard main page. The student has already been authenticated and is clicking to enter their fully online course. A big part of students having success in taking online courses is making sure that they continue to access the course. The data shows that QM courses have 10% higher courses accesses that non-QM courses.

Insight #4:  QM Courses had 12% higher course access minutes than non-QM courses

Students in QM courses are spending more time (12% more time) in their online courses than students taking courses that have not been QM certified. This refers to the time (in minutes) spent inside an online course. This could be because QM courses have met Standard 4, Instructional Materials. This standard checks that all course materials are contributing to the goals of the course.

Insight #5:  QM Courses had 51% higher course content item counts than non-QM courses

Nothing frustrates students more than logging in and seeing an empty course. This causes students to be apprehensive of what their experience will be like and start the course with a negative outlook. QM courses had a whopping 51% more course content items than non-QM courses. QM Standard 7 focuses on Learner Support. This checks the online course for clear instructions, descriptions, links to technical support and detailed explanations on how resources can help students succeed.

Insight #6: QM Courses had 58% higher assessment counts than non-QM courses

58% more assessment opportunities for students are available in QM courses vs non-QM courses. This number in Blackboard represents items such as tests, quizzes and assignments. This big difference is most likely due to QM Standard #3, Assessment and Measurement. QM certified courses track that online courses “provide learners with multiple opportunities to track their learning progress”. This puts more emphasis on spreading out assessments over multiple assignments and exams instead of having only a few high stakes assessments.

Insight #7:  QM Courses had 69% higher tool item count per course than non-QM courses

In Blackboard, the count of tool items refers to discussion forums, blogs, journals and wikis. A staggering 69%more tool items (on average, per course) are found in QM courses over non-QM courses. QM Standard #6, Course Technology, looks for “course tools that promote learner engagement and active learning”. Instructional designers frequently recommend using engaging tools like forums, blogs, journals, wikis and other tools to continuously increase participation and engagement for students.

Insight #8:  QM Courses had 7% higher marks on course evaluations than non-QM courses

Students enrolled in QM courses gave higher marks across the board, in every category than students enrolled in non-QM courses. Overall, 7% higher marks in QM courses. This table shows the sum of ‘Excellent’, ‘Very Good’ and ‘Good’ responses.

Conclusion

Overall, we found that courses that had received Quality Matters™ certification had higher points of student interactions, higher submissions, higher instances of course accesses, higher minutes of time spent in course, more items and resources to review, more opportunities to check their learning via assessments, more variety in terms of tools and provided higher course evaluations marks when compared to courses that had not received the Quality Matters certification.

The evidence suggests that online courses that have the QM certification are more effective and engaging.

Direction from Quality Matters Rubric
The Quality Matters™ rubric contains 8 General Standards and 43 Specific Review Standards. A scoring system and set of online tools facilitate the review by a team of Peer Reviewers.

Unique to the Rubric is the concept of alignment. Critical course components- Learning Objectives, Assessment and Measurement, Instructional Materials, Course Activities and Learner Interaction, and Course Technology – should all work together to ensure students achieve desired learning outcomes.

Appendix

Interactions
An interaction is defined as a user clicking into an item in Blackboard. Many types of student actions can lead to an interaction being logged, such as accessing a course item, submitting a discussion forum post, etc

Item Count (Assessment)
The count of assessments (e.g. tests, assignments, etc.) in Blackboard.

Item Count (Content)
The count of content items (e.g. pages, documents, etc.) in Blackboard.

Item Count (Tool)
The count of tools (e.g. discussion forums, blogs, journals, etc.) in Blackboard.

Submissions
A submission is defined as a student submitting an assignment, blog, discussion, journal, Safe Assignment, survey, and test submissions.

Average course accesses
This refers to a student clicking into a course from the Blackboard main page. The student has already been authenticated and is clicking to enter their fully online course.

Average minutes spent in course
This refers to the time (in minutes) spent inside an online course.

Course evaluations
All FIU courses include a student course evaluation at the end of the course. Students rate the following items as no response, excellent, very good, good, fair or poor:

  • Description of course objectives and assignments
  • Communication of ideas and information
  • Expression of expectations for performance in this class
  • Availability to assist students in or out of class
  • Respect and concern for students
  • Stimulation of interest in course
  • Facilitation of learning
  • Overall assessment of instructor

Thanks!

olanalytics@fiu.edu