Developing, implementing and improving learning outcomes assessment in Qatar’s public higher education

This study reports on the development and implementation of the Student Learning Outcomes Assessment process and the system implemented at Qatar University. This system consists of two phases, initiation and enhancement , with progressive and continuous improvement in the assessment efforts. The initial 2006–2011 findings suggest effective strategies that can be used to build and develop assessment culture and knowledge. This study is significant because it provides ways that other universities in the region can be engaged in building their quality assurance system and continuously improving student learning outcomes


BACKGROUND
In 1973, a teacher training college and the first higher education college was established in Qatar. Since then, Qatar University has been established and has grown tremendously; the university now offers thirty-six undergraduate programs, nineteen master level programs, two Ph.D. programs, one professional doctorate and four postgraduate degrees. With the goal of expanding and positioning itself as a model regional university, Qatar University planned and implemented university-wide assessment programs for quality assurance.
Over the past two decades, the increased attention to outcome-based education has motivated a more accountable system. 1 argue that outcomes can be used to define a process and its results. This approach is best described as result-oriented thinking, in contrast to the approach of input-based education. Due to the shift to outcome-based education, in the last six years, Qatar University has invested considerable resources, time and effort in establishing the academic program learning outcomes to improve student attainment. In addition, the quality assurance procedures adopted by Qatar University have provided the required evidence to assure stakeholders of the quality of education and learning experiences that the students receive and to demonstrate the added value and contribution of the university to the community at large.
The purpose of this study was to report on the conceptualization and experiences of a quality assurance initiative implemented by Qatar university. The study reports on the guiding principles, planning, piloting, implementation and refinement of the process of the Students Learning Outcome System at this Gulf University. Furthermore, the study provides insight into the challenges and lessons learned from the implementation of the Student Learning Outcomes Assessment process as part of the larger Quality Assurance Purpose and initiative that has been undertaken.

Outcome-based Education, Accountability and Assessment
Outcome-based education was referenced sporadically many decades ago and brought to center stage in the late eighties and early nineties. 2 Student learning outcomes are the culminating demonstration of learning. 3 Fundamentally, student-learning outcomes consist of the amount of student knowledge and ability level at the end of the course. This model of education has changed the educational philosophy of teaching and learning. 3 argues that there is a significant difference between outcome-based education and simply defining outcomes for an existing curriculum. Outcome-based education "does not mean curriculum based with outcomes sprinkled on top, it is a transformational way of doing business in education" (p. ii). 3 The emphasis on outcome-based education and the increasing demand for accountability and transparency by stakeholders, such as employers, parents, accrediting agencies and students on higher education institutions have forced universities to hold themselves accountable for the quality of the education offered. In addition, the intensified competiveness among these institutions on a local, regional and global scale has required the university to provide evidence that students are learning the established program objectives.
Moreover, with global competitiveness in higher education, universities have a small chance to survive and to continue to secure funds and stakeholder support without providing evidence-based measures that the universities are offering high quality learner-centered education. Thus, the assessment of Student Learning Outcomes (SLOs) has become an increasingly important tool to measure educational effectiveness. 4 More importantly, this assessment serves as a valuable mechanism that transforms and reorients the focus of education from a teacher-centered to learner-centered model. 5 -11 With the increased focus on and demand for university accountability, research on the effects of SLO on education and curriculum improvements has grown tremendously. Significant evidence from the existing body of research literature has demonstrated that developing and writing student learning outcomes ensures the attainment of the expected skills prior to student completion of the graduation requirements from an academic program and increasingly optimizes the implementation of pedagogies and learning strategies for student educational goals. 12 -21 With this consideration, an increasing number of universities are continuously assessing their students' educational gains and learning outcomes using multiple internal and external, direct and indirect assessment tools. Furthermore, the movement toward transparency and accountability has forced higher education institutions to publish their assessment results of student learning outcomes and to make them available to the public and the community at large. 11,22 -27

PHASES IN THE DEVELOPMENT OF THE TWO KEY PHASES
Qatar University's Students Learning Outcomes System was developed using two key phases. The first phase, known as the initiation phase, consisted of preliminary development and piloting of the learning outcome assessment process. Piloting of the learning outcome assessment process was based on a specific number of programs offered at the university. The second phase, known as the enhancement phase, consisted of the enrichment of the SLO process and implementation in all programs offered at the university. To provide readers with a structured summary of these two phases, an overview of the various stages that are subsequently discussed in further detail is provided in Table 1.

THE INITIATION PHASE Overview and guiding principles
The United Nations Development Program's report on the Quality Assessment of Education Programs in Arab Universities provided various recommendations for institutions selected by the United Nations Development Program (UNDP) to participate in its 'Enhancement of Quality Assurance and Institutional Planning' project at the Regional Bureau for the Arab State. For Qatar University, the recommendations were tracked and implemented. 28,29 In 2005-2006, Qatar University established a Quality Assurance Group that subsequently metamorphosed into the Office of Academic Evaluation. In 2006-2007, the Office of Academic Evaluation (OAE) was established as a part of the reform project and is recognized as an Internal Quality Assurance Unit responsible for the development and implementation of the student learning outcome assessment process and for other quality assurance processes, including academic program reviews and administrative units' periodic review processes. These elements are essential to ensure the quality of QU academic offerings and to ensure that the competencies of QU graduates are aligned with labor market outcomes.
In 2007, the OAE developed a Student Learning Outcomes Assessment System (SLOAS) to assess the program level student learning outcomes and to ensure the continuous improvement of academic programs offered at the university. The system was created to provide a platform through which stakeholders and the larger community can understand the mission and objectives of each academic program offered at the university. In addition, the expected students' skills and abilities attained upon graduation from each of the university programs were made open to the public. The system was designed to be a self-diagnostic tool that can provide academic programs with opportunities to identify curricular and/or teaching and learning issues and to identify appropriate strategies to improve the programs. In the Initiation Phase, the following plan was utilized and included the three following elements: 1. The phase included every academic program offered at the university, except those colleges accredited or seeking accreditation by an international professional accrediting agency (College of Business & Economics: The Association to Advance Collegiate Schools of Business AACSB, College of Engineering: Accreditation Board for Engineering and Technology ABET, and College of Education International Recognition in Teacher Education IRTE). 2. The system mandated the definition of a mission statement, well-articulated educational objectives and measurable program level student learning outcomes. 3. An assessment plan was developed and the assessment results were used to improve programs.
To ensure successful and consistent implementation, the system incorporated professional development programs for the "owners" at each step of the assessment process. The professional development programs were designed to instruct the faculty in best assessment practices and to build an institutional capacity in the area of student learning outcomes assessment.
The main goals of the SLOAS were to align program educational objectives to QU's mission and overall strategic objectives and to ensure continuous improvement based on reliable evidence. The SLOAS provided mechanisms to ensure that the program-level students learning outcomes were clearly stated and measurable.
The SLOAS was introduced to the university community via various workshops, where the roles and responsibilities of the colleges, departments and faculty members were clearly explained. All programs offered at the university, except for programs that were preparing for accreditation at that time, were required to comply with the SLOAS system requirements and to implement the assessment process. Findings and Discussion † Focus on learning outcomes rather than learning processes. † Review and rewrite student learning outcomes statements. † Complement the use of internally developed assessment tools with the use of standardized assessment tools to provide confidence in the validity and reliability of the assessment results by comparing results obtained using both tools.

Enhancement Phase Stages
New organizational structure † APLOA: the Academic Program Review & Learning Outcome Assessment Office was established in 2009 † Student learning outcome assessment committees at the program, college and university levels were established. † APLOA created the positions of assessment coordinators at the program and college level.
Developing the process Automating the Assessment Process and reporting † Development and implementation of an Online Assessment Management System (OAS) aiming to facilitate the institution's systematic collection, use, and review of information about educational programs to improve student development and learning. † Mandating all programs to use the automated assessment system † Extracting assessment reports at the program, department, college, and university levels.
Colleges and programs that were temporarily exempted from complying with the SLOA system requirements included those that were participating in the North American Accreditation process. The decision not to engage these programs and colleges undergoing their program accreditation was based on the fact that they were implementing their own assessment processes in compliance with the requirements of the accreditation agencies.

Development of Student Learning Outcomes Assessment System
SLOAS development was based on three quality assurance stages: (1) Planning Stage, (2) Implementation Stage, and (3) Development and Quality Assurance Stage. The system recommended an implementation cycle spanning two academic years, during which each program was required to assess all program-defined student-learning outcomes. At the end of an academic year, each academic program was required to submit a report on its assessment activities to the leadership at each college i.e., to academic deans as well as to the OAE. The Planning Stage. This stage included four parts: First, the program's vision and mission needed to be written with an emphasis on the purpose and important contribution to the society and targeted audience. Second, the program objectives and student learning outcomes were required to be defined with a clear articulation of the knowledge, skills, attitudes and values that students were expected to acquire by the end of the program. Third, each program was required to ensure that the level of Student Learning Outcomes satisfied the following criteria: 1. Linked to the program mission and objectives 2. Clearly articulated 3. Measurable 4. Achievable after completing a number of purposeful educational activities 5. Describing student ability upon graduation from the program Finally, the planning stage included the development of an assessment plan with a biannual reporting cycle consisting of mapping between the program mission, objectives and outcomes, in addition to mapping student learning outcomes to the curriculum. Furthermore, a detailed reference of the various assessment tools linked to the courses in the curriculum was defined, including a timeline indicating when these tools were to be used during the assessment cycle. All plans included program performance expectations or benchmarks for each of the defined student learning outcomes.
The Implementation Stage included a timetable for the assessment plan. The SLOAS provided a list of guidelines for the implementation of the assessment process at the program level. These were as follows: . Identification of appropriate assessment tools to measure the specified SLOs. . Identification of the entity (faculty/committee) responsible for collecting, analyzing and interpreting the program data and information from the implementation of the assessment tools. . Determination of the entity (faculty/committee) that would write the final results report. . Determination of how assessment results would be utilized.
The Development and Quality Assurance Stage. This stage included data collection, analysis and validation, in addition to the adoption of actions to improve the program and enhance the quality of the education offerings. Furthermore, clear guidelines and templates were provided to the owners. In particular, the SLOAS recommended the use of two assessment tools for each learning outcome to provide a strong valid and reliable evidence of the assessment results, with at least one of the assessment tools consisting of a direct measure of student performance.

The Implementation Processes
During the first year (2007/08) of the two year-cycle, university programs produced documents clearly articulating their vision, mission, objectives and student learning outcomes. During year two of the process, these programs were required to design assessment plans and assess the students' attainment of the declared program-level student learning outcomes. Each program was entrusted with the implementation, data collection, analyses, interpretation and reporting results and recommendation on the use of assessment results to improve the academic offerings.

Final Steps in the Initiation Phase
The Office of Academic Evaluation was responsible for reviewing the annual learning outcomes assessment progress reports submitted by all programs. This process included a review of the program mission statements, educational objectives, and student learning outcomes. The OAE also reviewed the mapping of the learning outcomes with different levels of thinking and cognitive domains introduced in Bloom's Taxonomy. 30 -32 In addition, the OAE examined the alignment of the program objectives with the program mission, student learning outcomes and validity of the assessment methods and tools in measuring the student program learning outcomes.
The Office of Academic Evaluation review Attiyah and Khalifa 29 indicated that during the initiation phase, only 22 academic programs submitted SLOs progress reports. Of the 22 programs that submitted the SLOA reports, 14 programs had mission statements that were succinct, realistic, and current. At least 8 of the 22 programs had clearly included their stakeholders in their mission. The OAE report demonstrated that the SLOs of 12 of the 22 programs were clearly articulated and mapped well to Bloom's cognitive domains. These SLOs were consistent and supportive of the academic programs educational objectives. The assessment methods and instruments developed by 11 of the 22 programs were reported to be aligned with what the students intended to learn.
Based on these above findings, the Office of Academic Evaluation recommended that all academic programs must focus on learning outcomes and student attainment, must review and rewrite their mission statement and student learning outcomes statements and must complement the use of internally developed assessment tools with uniform formatting in both direct and indirect ways. The validity of these measures was confirmed through a strong association between the measures. 29,33 THE ENHANCEMENT PHASE New organizational structure Although the initiation phase was a pilot project, the results were promising. It was evident that there was a need to increase the visibility of all aspects of the process and to restructure the overseeing authority in such a manner that SLOAS became 'built into' the organizational structure of the university. An office was established within the university, and an assessment coordinator in each college supported the work of the program assessment committees. Their role was to oversee the learning outcome assessment processes and to follow up on the implementation of the adopted improvement action. This process demonstrated evidence of continuous efforts in strengthening the culture of the assessment at QU by further improving the assessment process and outcomes and improving the involvement and contribution of a larger number of the university academic staff. During this Enhancement Phase, 44 programs participated in the learning outcome assessment process. Thus, the committees at the program, department, college and university levels were essential in demonstrating the university's commitment to support the faculty and build faculty capacity in areas of assessment.

Developing the Assessment Process
During this enhancement phase, the SLOAS was further refined, and a more comprehensive process was developed to improve quality assurance and continuous improvement system; this process was before shaped using the Six Sigma approach to Quality Assurance. 34 The process was built on the Six Sigma DMAIC Methodology (Define, Measure, Analyze, Improve, Control). 35 -41 The five phases of the Six Sigma DMAIC methodology were adapted to the student learning outcome assessment process created by the university by: (1) Defining and Planning phase; (2) Measuring phase and The Data Collection and Reporting phase; (3) Analysis and Evaluation phase; (4) Continuous Improvement phase and finally (5) Control and Feedback loop. 42 Refining the Assessment Process During the 2008-2009 academic year, nine academic programs were selected and requested to implement the enhanced SLOAS. In addition, they were required to submit an annual student learning outcomes assessment report by the end of the academic year. Each selected program was not preparing for accreditation or did not identify an accrediting agency at that time. These selected programs included the following: College of Arts and Sciences programs in History, Geography/Urban Planning, English, Sociology, Arabic, and Arabic for Non-native Speakers programs; Islamic Studies including Sharia; and the Law program offered in the College of Law.
Reports submitted by the selected programs showed great variation, particularly in the achievement of the target or performance indicators. They also varied in the analytical method and reporting the results and improvement actions. However, it was clear that more work is needed to further define and communicate the assessment related processes and to develop the appropriate mechanisms and tools to support the assessment related activities within the university. In particular, the control feedback loop (see Six Sigma DMAIC principles), where the assessment results are to be used to improve the quality of the academic programs, remained a challenging task. Nevertheless, the second cycle of the 2008 -2009 student learning outcomes assessment reports submitted by the selected programs indicated that all selected programs achieved significant progress in integrating students learning outcome assessment in their day-to-day teaching and learning activities.
The variation in the implemented processes and method in selecting samples of assessed students indicated that the assessment process required further refinement to ensure the reliability and validity of the assessment results. The university-wide Students Learning Outcomes Assessment Committee was established and was responsible for examining the assessment reports submitted by the selected academic programs. This body had concluded its review report by stating that "Continuous effort is needed to refine the assessment related processes and to implement them at all programs offered by the university as well as to promote a culture of assessment and quality assurance within the university community". 43 The report emphasized the need for a university-wide assessment process as a key element in the continuous and effective improvements of academic programs.
The APLOA went further to initiate professional development activities in the system and invest extensively to ensure a more comprehensive understanding of the objectives of the student learning outcome assessment process, as well as to incentivize the academic staff to increase motivation and build a sense of process ownership. For example, the office provided conference support, a Best Assessment Report Award, and funding support for assessment-related professional development.
During 2009-2010, the second year fine-tuning process was based on the findings of the previous year, where a refined SLOAS was introduced to all academic programs offered at the university, and all programs were required to implement the system and submit an annual assessment report. Specific guidelines were provided to aid in understanding the new system requirements, and a number of templates were developed to ensure that a uniform university-wide reporting mechanism was in place. In addition, at the end of 2009 -2010, several learning outcome assessment experts from various international universities with extensive experience in developing and monitoring institutional and program accreditation processes were invited to review the annual assessment reports submitted by the programs. They were asked to provide feedback and recommendations for improving the process and the assessment activities implemented. The reviewers were asked to respond to the following questions as they reviewed the reports: 1. Do the selected assessment tools measure the students' learning outcomes? 2. Are the recommendations aligned with the results? 3. Are the action plans aligned with the recommendations?
Acting on the recommendations of the external review team, APLOA developed a professional development program to inform stakeholders of the enhanced assessment process. This professional development program was also part of Qatar University's efforts to improve the adopted student learning outcomes assessment process and to reinforce the culture of evidence based on continuous improvement. This implementation of SLOAS for two year-long cycles (2008 -2009 and 2009 -2010) was helpful for APLOA in refining and adjusting the assessment process to ensure effectiveness, efficiency, and sustainability. More importantly, feedback collected during these two years indicated the need for a reduction in the number of Student Learning Outcomes that was annually assessed.
APLOA further improved the assessment process guidelines and reviewed the reporting mechanism. The 2010-2011 annual assessment reports submitted by programs included the following: Automating the Assessment Process and reporting The continuous drive to improve the SLOAS resulted in the development and implementation of an Online Assessment Management System (OAS). This system supported and facilitated the assessment process. The OAS is an electronic system that was designed to allow programs, departments, colleges and institute to collect, manage, and report data related to student learning outcomes assessment for continuous improvement purposes.
The OAS was fully deployed at the start of the 2011 Fall semester. All programs offered at the university, excluding programs at the College of Education that used an e-portfolio-based software product, were required to use the system to record the program mission, objectives, learning outcomes, curriculum maps, assessment plans, details of assessment activities and assessment results. The OAS enabled programs to record assessment activity details, including assessment tools, student work and rubrics and their scoring student and data from multiple assessors. The system recorded the data to generate a number of reports designed to aid in the analysis of the results. More importantly, the OAS simplified the assessment process by substantially reducing the high workload associated with the data collection and reporting steps and error risks during the data collection and aggregation steps. Furthermore, the university invested time and resources in offering over 100 comprehensive professional development sessions on OAS for coordinators, faculty and staff in all programs and colleges. Other initiatives included specialized professional development activities on assessment processes and the best practices on the use of the assessment results dedicated to improving the quality of academic programs.

LESSON LEARNED AND CONCLUSIONS
Key elements necessary for sustainable and successful implementation of SLOAS included the following: continuous institutional support, faculty involvement, and effective and sustainable communication and feedback between all stakeholders. Thus, it was important to create opportunities for faculty engagement in the assessment process at the institution level to further motivate faculty members and staff to adopt the process. This was an essential element to provide legitimacy to the system and to establish a culture of assessment and continuous improvement within the university.
The system is expected to improve teaching and learning, with the goal of improving student performance, achievements, and competencies to meet the expectations, needs, and requirements of the national, regional and international job market and the program stakeholders. Programs follow the best practices and provide evidence of student performance and achievement on the assessed outcomes, as well as an analysis of the trends and cohort-based analysis. Data aggregation of both the formative and summative assessment findings, as well as the indirect and direct assessment findings, were carefully collected and analyzed. Assessors then integrated the results of the different assessment types into a narrative to reach conclusions that were reliable and consistent to assure evidence-based continuous improvement.
Ensuring the sustainability of SLOAS is critical and decreasing of the number of students learning outcomes is imperative to retain the momentum of the assessment process at the university. Programs should focus on assessing what is fundamental to the program's mission and educational objectives. Thus, the programs were recommended to focus on reviewing the articulation of the SLOs and to have multi-year assessment plans to ensure that every outcome is examined over a reasonable period of time. The introduction of such measures has, in our experience, considerably decreased the workload and complexity of the assessment process.
Importantly, Qatar University has exerted a large effort to implement a student learning outcome assessment process in a rather short period of time. As is the nature of any quality assurance system, there is room for improvement not only to ensure the sustainability of the process and the positive effect on teaching and learning but, more importantly, on the achievement outcome of the university graduates because there is little empirical research examining the effect of assessing student learning outcomes on actual student learning and achievements. 24,44 -49 Studying the effect of assessing student learning outcomes on student achievements provides valuable evidence to students and stakeholders such that the university adheres to its own goals and mission. 50 -54 This study would be the next step for this development work and evaluation.
However, there is a need to further improve the process in many areas, particularly the analysis, control and feedback aspects. Validation and reliability of the assessments requires further attention and integration in the system. The implementation of the SLOAS has generally improved when time and effort was spent on examining the quality of the collected evidence, assessment tools, recommendations and adopted improvement actions. Faculty buy-in and involvement in the assessment process were also evidence of cultural changes that were vital for a sustainable Students Learning Outcomes Assessment System. Furthermore, within the Six Sigma DMAIC framework, it was vitally important to strengthen the data analysis methodology and the interpretation of the results to identify how the results could be translated into decision-making and strategic planning. In addition, to ensure consistency of the rating, inter-rater reliability rating coefficients should be calculated for the adopted rubrics. 55,56 These actions will increase the level of trust in the findings and ensure the validity and reliability of the results and consequent actions.
Finally, the continuous support and investment in faculty professional development related to the outcome assessment with efforts in providing an open environment for faculty members to share and exchange experiences and to discuss and learn the best practices will significantly improve the overall assessment process. This outcome may be achieved by offering programs and assessment forums, faculty contributions and an understanding of the importance of the assessment. Building the capacity and providing faculty with the tools and support needed to conduct the assessment would be logically integrated into the university strategic priorities. This goal requires the university to invest significantly in developing expertise in the assessment of learning outcomes and continuous and ongoing faculty engagement to sustain quality assurance initiatives. The assessment of student learning outcomes could be a key element of the university quality assurance system, with a focus on evidence-based improvement of academic programs.
Special attention could be given to subjects and programs where no accreditation agencies exist. In addition, it is important to involve program stakeholders in the assessment process to ensure the effectiveness and to best prepare the university graduates for their future careers or postgraduate studies. Stakeholder participation and involvement will enable academic programs to incorporate stakeholder-desired learning outcomes from the outset and permit interests to be integrated within the university curriculum.
Overall, one can easily argue that quality assurance and the student learning outcomes assessment journey at Qatar University are taken very seriously and are positioned high in QU strategic priorities. The university's large investment in the assessment process illustrates that the quality of educational offerings and the competency of the university graduates are central to the university mission and are rightly positioned at the top of the university strategic priorities.