Evaluation Research in Government

I have spent the past two days with a group of dedicated professionals from many backgrounds developing SMART objectives for a monitoring, evaluation and accountability plan.  Snug in the surroundings of beautiful trees, rock and wildlife,  away from the sounds of civilization, a group of multi-discipline professionals have begun the process of what will eventually be evaluation research.  The government has put a new emphasis on evidence-based decision making.  Every new program, project or strategic framework requires a monitoring, evaluation and accountability plan.  This plan clearly lays out specific, measurable, achievable, relevant and time-bound (SMART) objectives that progress and results that can be evaluated against on a regular basis.

Anderson says, "More emphasis should be placed on developing a sound evidence base for a policy through long-term impact evaluations of policies and programmes. It is argued from a realist position that such evaluation should be theory-based and focused on explaining and understanding how policies achieve their effects using ‘multi-method’ approaches." (2002, abstract)

This new process is a shift from evaluation research, where professional evaluators were contracted by government to evaluate programs, policy, strategic framework.  This evaluation took place at the end of a project with no thought given to what the professionals were measuring. The original process was quite often an exercise in frustration for program staff and evaluators, where no real results could be reported.  

Currently in most departments of the Government of the Northwest Territories, much time and effort is put into thinking about the evaluation measures that will be incorporated into an evaluation framework for new projects.   This new way of working is a positive move; that ideally will result in the culmination of promising practices in both evidence-based decision making and program design.  Only time will tell, but I am hopeful




Anderson, I. (2002), Evaluation, Policy Learning and Evidence-Based Policy Making. Public Administration, 80: 1–22. doi: 10.1111/1467-9299.00292

Me, A Researcher?

When I began this journey in the Masters in Learning and Technology, the part I most dreaded was research.  I had no idea how research had evolved, or how little I knew about research.  I was stuck in the mire of research courses from over twenty-five years ago.  Quantitative research with reams of numbers that left you limp and lifeless.  These numbers invaded dreams and waking life.

Imagine my surprise when my textbook, Mindful Inquiry in Social Research showed up in my mailbox.  I loved the title, and I couldn't wait to dive into the book.  And dive into it, I did.  It became my lunchtime and bedtime reading.  I couldn't put it down!  Statements like, "our philosophy of research, which we call mindful inquiry, is a synthesis of four intellectual traditions phenomenology, hermeneutics, critical social science, and Buddhism." My eyes were drawn to critical social science and Buddhism. I asked myself, how could a textbook on research include faith?  Not my faith necessarily, but I was drawn to this idea.  I am also a proponent of critical pedagogy, and I was convinced that critical social theory and critical pedagogy had to be linked somehow.  And in fact, they are!

This was an eye-opening experience.  I soon learned that research as I had imagined it, was not necessarily the only choice.  That there were many cultures of inquiry and many ways to shape and conduct research that focused on people and real community issues.  Research that's goal was to work towards a more socially just society, not just through gathering information, but turning that information that was people and issue focused into action.

My mind and my heart are now open to the idea of conducting primary research to improve the society in which I live.  My jumbled head sees potential research questions everywhere, in violence against women, women's equality, education, assistive supports for people with cognitive challenges, government policy, and legislation.  This world in my mind is endless and exciting.  A world that I hope someday to be able to be an active participant in creating knowledge and action towards social change that makes a difference for people.

Bentz, V.M. & Shapiro, J.J. (1998).  Mindful inquiry in social research. Thousand Oaks, CA: Sage.

Finding my academic voice

Part of being an academic is feeling confident in the knowledge you have and sharing it in many different forums.  Student blogs are one way to begin this sharing journey.  Nackerud, S. and Scarletta, K. said, "many people in academia view - and judge-blogs as forms of academic production or a vehicle for scholars to become public intellectuals." (2008, p.71)   I am an infant in this journey.   I am tentatively spreading my wings, but I am not quite ready to fly. Does this mean I don't have anything to say? No! Like many, I have interests and opinions on many things.  However, I am not far enough along in my academic studies to be able to concretely back up my opinions with original research.

Estes says, "Blogging helps me to bring together aspects of my academic and non-academic lives, as well as to start thinking about issues that will eventually get my academic attention." (2012, p. 974) This statement speaks directly to me.  When I get excited about an idea, technology,  or a new way of thinking about things, my blogs reflect that excitement and passion. I experiment with ideas and how I can use these ideas to enhance my practice in adult basic education. I see possibilities and write about these possibilities.  These are informal musings that may some day lead to direct research. Currently, they are ideas; ideas that raise my interest and enthusiasm for making the world, my practice and adult basic education more accessible and just. This is my academic voice, a voice of possibilities for change in myself and my practice.

Estes, H. (2012). Blogging and academic identity. Literature Compass, 9(12), 974-982. DOI: 10.1111/lic3.12017

Nackerud, S., & Scaletta, K. (2008). Blogging in the academy.  New Directions for Student Services (124), 71-87.  DOI: 10.1002/ss.296

Pondering a Research Question

In qualitative research, Agee says, "A broadly framed question can serve as a basis for initial and emerging sub-question." (p. 434)

Agee goes on to say, "A single overarching question allows a researcher to capture the basic goals of the study in one major question. A clearly stated overarching question can give direction for the study design and collection of data and offer the potential for developing new, more specific questions during data collection and analysis. Sometimes, these broader research questions are not stated as questions but.rather as goals for the study." (p. 435)

This information is food for thought.  My research question for my capstone research paper is quite specific, and isn't written in question format. It is:

The purpose of this study will be to understand how to transition from a classroom-based learning approach to an online or blended learning approach for adult learners in the NWT. This study aims to identify and understand the challenges and opportunities faced by adult learners enrolled in adult basic education in the NWT, transitioning to online learning.

The demographic of the learner that we anticipate transitioning to online learning is:

• Adult learner enrolled in adult basic education (K-12 for adults);

• Predominantly First Nations and Inuit;

• Age range from 20-60 years; and

• Live in small northern communities ranging in size from 60 residents to 6,000 residents.

Courses are delivered in English.

So this leaves me wondering, do I begin with a question like; how do we transition to online learning in adult basic education in the NWT?  Or What are the benefits and challenges of delivering online or blended learning in adult basic education in the NWT.  Both of these questions are broad, but they certainly don't have the depth of the original description and in some ways change the intent of the research.

Writing research questions is an art form, one I think I need to ponder more.

Agee, J. (2009) Developing qualitative research questions: a reflective process, International Journal of Qualitative Studies in Education, 22:4, 431-447, DOI:

10.1080/09518390902736512

Auditory learners and research

I began an adventure this week, looking for supports for my son who is on the autism spectrum.  He is considered moderate. However, having said that he is very intelligent and is beginning a new college program in the fall.  Through some support, we discovered Livescribe pens.  These pens allow learners to create pen casts through tapeing either their voice and thoughts or a lecture and taking key word notes.  We received the two pens, one for me, and one for my son because I need to be able to use the technology to support him. I started to experiment with the pen in my Introduction to Research, class work. I am currently critically analyzing six research papers.

As an auditory learner, I find it somewhat difficult to live in the online world.  I spend a lot of time in my home office talking to myself about information and knowledge that I have read either on discussion boards, blogs or class readings.  If anyone heard me, I am sure I would be committed.  What I discovered while experimenting with the Livescribe pen is that this will also enhance my learning in my Masters program.  I can read, say thoughts out loud while writing notes. I can then sync it to my computer or handheld device, and I have a pen cast of my thoughts and my notes together, which I can listen to and read over and over.  Quite often when I read information, I take notes and I speak out loud to myself, but because my brain is trying to do too many things at once, I lose my verbal thoughts before I can get them down.  This technology enables me to capture both at the same time and write down key words, speak the thoughts around the key words and capture it all at once.  Brilliant!

I have done a pen cast that I am attaching here.  I am still new to the technology, so it is not polished yet, but it will get better and better.  I have also attached youtube videos about the Livescribe pen in hopes that you may be able to use this technology with your learners who may be struggling or yourself in your own endeavors.

Page 3 - LS3 LTR Single Subject Notebook 1

Page 3 - LS3 LTR Single Subject Notebook 1https://www.evernote.com/shard/s635/sh/13089ccb-7267-42b1-94ec-e5f62fde878f/bf6799ac18b5f02bbf08f166327a2669

https://www.youtube.com/watch?v=20Hgc_B4xuM

Towards Excellence

Today I am going to talk about teamwork in academia. The thought of trusting a team of people I do not know and have never met to impact my work, and my mark initially was terrifying.  It kept me awake at night.  I felt a bit powerless to influence what my work would be and would look like.

I work in teams all the time at work. Some of these have been teams with synergy and focus with people who behave like a caring community.  These teams have a passion for the work and each other and strive to ensure that all are heard. Others have been working groups where everyone is interested in themselves only, with little synergy and a lot of in-fighting.

So now imagine that you are challenging yourself to do a Masters program in something that you feel less than capable in and you are assigned to teams with complete strangers.  Frightening! At best you imagine them as working groups with people only interested in their voices. Right?

Wrong! 

They are TEAMS!   The teams I have been blessed to work with are caring people who demand excellence of everyone, involve everyone in the process and continue to create quality work.  I have learned more about the team process in the five months I have been in the MALAT program than I have in 30 years of work.  This learning is beyond the subject matter.  It's about accountability; it is about learning, and it is about excellence.  This has been a blessed experience that I will take to my workplace and utilize the lessons learned to build strong, passionate, accountable teams that work together to improve education in the Northwest Territories.

To Delphi or not to Delphi?

I find myself fascinated by the Delphi survey technique in research. This technique enables a researcher, who plays the role of facilitator to work with a small group of "experts" to ideally reach consensus on an issue.  Research Areas in Distance Education:  A Delphi Study used this approach to reach consensus amongst a group of experts in distance education about what areas of research in distance education are most needed and most neglected.

Further reading on Delphi survey techniques pointed out that this can be an efficient way to get consensus on an issue; however, there are a number of challenges involved in ensuring that the research study is reliable and valid.  Research guidelines for the Delphi survey technique pointed out that, "Uses systematically and rigorously, the Delphi can contribute significantly to broadening knowledge... However, careful thought must be given before using the method; there are key issues surrounding problem identification, researcher skills and data presentation that must be addressed." (Hasson, F, Sinead, K, McKenna H., 2000. p. 1008)

The strengths of this survey technique, however, would benefit the NWT in its move towards developing and delivering adult basic education using a distance delivery model.  One of the key challenges in doing this will be getting adult educators and instructors buy-in for the change.  This technique could be used to find out what key supports that adult educators would require to feel comfortable with this change.  There are approximately 75 adult educators in the NWT who deliver government funded adult basic education programs.  That is too many to be directly involved. However, criteria for participants could include:

• must have been an adult educator in the NWT for a minimum of five years;
• must not be a program manager (they do not do direct delivery)
• must have a range of participants from each of the five regions.

The utilization of this selection criteria would mean approximately 25 participants would be involved.

The initial open questions could include; what supports and what changes would need to be made to the classroom space.

"The classic Delphi technique had four rounds." (Hasson, F, et al., 2000. p. 1011)  The results of each round are reported back with clarifying questions in each round.  This could lead to consensus on priority supports and changes required to effectively implement distance delivery. However, Hasson (2000) does caution anyone using this method by stating, "it is important to note that the existence of a consensus does not mean that the correct answer, opinion or judgement has been found." (p. 1013) However even with the challenges and limitations of the Delphi technique, it would be a positive first step in moving this initiative forward.

Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing 32 no. 4: 1008-1015. CINAHL Complete retrieved from EBSCOhost.

Zawacki-Richter, O. (2009). Research areas in distance education: A Delphi study.  The International Review of Research in Open and Distance Learning, 10(3).