My name is Tim McCallum [1]. I work closely with Professor Jim Taylor [2] designing and building OER course offerings at the University of Southern Queensland (USQ) using the Context, Input, Process and Product (CIPP) evaluation model. The OERu context evaluation study conducted by the Australian Digital Futures Institute at USQ was based on the CIPP model to inform the planning for the prototype phase of our collaborative initiative. 

Guest post from USQ:  Open Education Week 2014. 

CIPP model for evaluation

In this blog post I would like to discuss the CIPP model for evaluation (Stufflebeam & Shinkfield 2007), but more importantly how we are using it to build our OER courses here at the University of Southern Queensland.  Joining the Open Education Week 2014 celebrations, we recorded a video signpost summarising the CIPP model and released this under an open license which will hopefully be reused in one of the OERu course developments. The CIPP model provides four evaluation types which are as follows:

Context

Firstly, Context, for example assessing the user’s needs and identifying opportunities or problems relating to particular users or groups (a little more on this later).

Input

The second evaluation type is Input.  This relates to the design aspect of how knowledge is acquired by the user. Some courses lend themselves to a more social or open design where micro-blogging or social media play a big part in the learning experience, personal mentoring is another consideration here. Perhaps the most important factor to consider when looking at this evaluation type is the software platform or delivery mechanism for the course in question. If we are designing the course for engagement, we need to be sure that the technology is able to support this.

Process

Process is the third evaluation type. This relates to the actual implementation of the course and its related activities. We, will be focussing on what learners actually do during the course.

Product

Finally Product; Product in a nutshell relates to return on investment, comparing actual outcomes against efforts made.

Putting CIPP into practice

Just to elaborate further on the first context evaluation type and to pull this all together, let's say we perform a survey at the start of an OERu course where a given number of users identify their intention to gain full credit for the course. If we have a KPI relating to this metric, we can make this part of the CIPP model for evaluation. We can combine survey results with empirical data or automated analytics (data mined from an OERu course) then use this data to derive information and make decisions which improve the KPIs and ultimately the success of future course offerings. Essentially we are creating an opportunity to find out what is working, or more importantly what is not working for particular users or groups. Once this information is derived, we can improve the course with a degree of certainty.

We are focusing on improving user grades and creating independent self-directed learners. We are working to improve user retention and we consider future enrolment and further engagement a sign of successful implementation.

Software platforms

I recently spoke with Professor Ken Udas [3], Professor Jim Taylor [2] and Dr Wayne Mackintosh [4] about delivery mechanisms for the OERu. This conversation took place shortly after I had completed the Open Content Licensing for Educators course (an OERu course delivered on the WikiEducator platform).

We discussed the wiki environment as an authoring platform and commented on its usefulness around version control, co-authoring and collaboration.

The Wordpress platform was introduced as a potential delivery mechanism. Wordpress offers many advantages in this arena such as responsive design templates and Wordpress themes as well as plug-ins, all of which allow rapid development and reuse.

Jim Tittsler and I have been discussing a WordPress plug-in called wiki-embed which lets you embed MediaWiki pages into your Wordpress site. I am in the process of installing and testing this plug-in.

A colleague and PhD candidate, Neil Martin [5] and I are also working together. Neil is investigating how User Experience design techniques could potentially improve the learning experience within the OERu. 

Be sure to keep a keen eye on the Google Summer of Code (GSoC) because the OER Foundation has been accepted as a mentor organisation for 2014. Perhaps one of the GSoC students will pick up on one of the software development ideas to refine and improve the OERu delivery model. 

Certainly a lot happening in this space, please leave any questions or comments on my blog below and enjoy open education week.

Notes

[1] Senior Analyst and Demonstrator at the University of Southern Queensland

[2] Board of Directors of the OER Foundation and Emeritus Professor for the Australian Digital Futures Institute (ADFI) at the University of Southern Queensland

[3] Deputy Vice-Chancellor, Academic Services and Chief Information Officer (CIO) at the University of Southern Queensland

[4] Founder of WikiEducator,  founding Director of the OER Foundation and UNESCO-COL Chair in OER at Otago Polytechnic

[5] Learning Technologist for the Australian Digital Futures Institute (ADFI) at the University of Southern Queensland

References

Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications (pp. 240-241). San Francisco, CA: Jossey-Bass.

About the author

Timothy McCallum
Senior Analyst - Demonstrator
The University of Southern Queensland 
Ph: +61 7 4631 1501 
Email: mccallum@usq.edu.au

LinkedIn

Blog