Strategies to Make Program Assessment Simple in a Digital Era: A Case Study

By: Dr. Edilberto Arteaga-Narvaéz

Dr. Kenia A. Parga Rivera, and

Dr. Lillian Gayá González

Inter American University, Metropolitan Campus

 

 Strategies to Make Program Assessment Simple in a Digital Era: A Case Study

Abstract

The assessment process of an undergraduate program is used as a case study to share some strategies to simplify the assessment process during a period where faculty members initiated the use of Tk20.  Some strategies to establish an organized, and systematic academic program assessment process for the development and improvement of the academic program while implementing TK20 are identified. The strategies identified may help others to move forward in the assessment process and make it permanent and sustainable.

Introduction

The Metropolitan Campus (MC) is the largest academic unit of the Inter American University of Puerto Rico (IAUPR). IAUPR is a private, Hispanic-serving institution with nine academic units or campuses and two professional schools (School of Law and School of Optometry). MC was established in 1960. It is licensed by the Puerto Rico Council of Education (PRCE) and accredited by the Middle States Commission on Higher Education (MSCHE). It also has specialized professional accreditations for several programs. MC offers 106 higher education programs.

Since 2003, assessment became a priority at MC, yet by 2013 the assessment process was still in a beginning phase (Self-Study Report, 2013). Several actions were taken to ensure a systematic, continued and sustained assessment process.  Among others, the level of responsibilities on assessment was clarified.  To support academic unit in 2010, IAUPR acquired the Tk20 platform for the management of institutional and academic assessment. Tk20 platform implementation is coordinated by the Associate Vice President for Student Affairs of IAUPR, who is the Unit Administrator. Among other responsibilities, the Unit Administrator provides campus access to the Tk20 account, produces reports, reviews information from academic campuses, trains the trainers (mainly, the Campus Administrators), and provides follow-up activities.

Tk20 is an integrated assessment planning and reporting system to collect and manage data to be used in institutional decision processes. It facilitates systematic data collection and generates detailed reports for accreditation compliance, program improvement, and the evaluation of institutional effectiveness. Data from several sources (departments and academic or service programs) are organized in a single location. It allows for demonstrated evidence on how the academic activities and student services contribute to the achievement of campus goals.

Several issues delayed the full implementation of Tk20 across programs in the Faculty of Sciences and Technology: among them, were a lack of a common language (program versus classroom assessment, evaluation versus assessment, metric versus grades), discrepancies in how to measure expected learning outcomes, and faculty members’ resistance.  This article explains five strategies adopted by the authors during the implementation of a Tk20 platform for the assessment of an undergraduate academic program and presents them as a case study to guide others.

Case Study

The Bachelor of Sciences in Natural Sciences (BSNS) program was licensed by the PRCE in 2012. The students’ competencies (Graduate Competence Profile) are listed in Table 1, below.

Areas Competencies
 

Knowledge

Describe the basic concepts of the natural sciences and technology.

Demonstrate the processes related to the administration, analysis and interpretation of data.

 

 

 

Skills

Critically evaluate a scientific article of a primary source.

Analysis, synthesize and communicate concepts effectively from a multidisciplinary point of view.

Compose written works using scientific information.

Use basic scientific equipment properly.

Carry out statistical analysis of experimental data and reach conclusions about these.

 

Attitudes

Recognize the impact of the natural sciences and technology by identifying their responsibilities, purposes and usefulness.

Table 1. Students’ Competencies (Graduate Competence Profile)

The program was conceived with an interdisciplinary approach.  A total of 120 credit-hours are required for graduation. These credits are distributed as follows:  48 General Education, 29 core courses, 31 major courses selected from a group of disciplines (Biology, Chemistry, Computer Sciences and Mathematics), and 12 electives.

Faculty started collecting data for program assessment purpose in 2014. At least, five strategies were identified during the BSNS program assessment process related to: duties at different MC assessment organizational levels, documents worked in advance, common language, clear standards, and information sharing. These strategies may contribute to the achievement of assessment tasks and emerge from the utmost lesson learned: To make assessment simple (Suskie, 2009; Walvoord, 2010).

Strategy #1: State clearly the duties for different assessment organizational levels

Academic program assessment, or simply, program assessment, is a faculty-driven activity. Professors are the members of the academic community best positioned to drive the assessment processes, to use results meaningfully in order to improve student learning, and to determine recommendations for academic decisions. However, administrative support is essential to achieve results and keep program assessment as an ongoing process. MC administrative representatives at different organizational levels are committed to the support of assessment efforts.

In order to ensure success in a systematic timely manner for the accomplishment of tasks at hand, duties at different Tk20 assessment organizational levels must be clearly defined and assigned (Monitoring Report to MSCHE, 2015). Accordingly, MC revised the academic assessment duties organizational structure by level, as described below, in Table 2.

Campus Administrator level

Dean of Academic Affairs
¨  Provide leadership for chairperson, faculty, office directors, and assessment coordinators

¨  Oversee that the approved mission and goals of the academic unit are incorporated into Tk20

¨  Oversee goals and objectives of academic programs

¨  Generate different types of reports available in the system, as needed or requested

¨  Help design and coordinate assessment training activities (to academic deans, chairpersons, and assessment leaders) across programs

¨  Assure compliance with campus and IAUPR norms and regulations, as well as with licensing, and accrediting agencies standards

Assessment Monitor                                                                                                                
¨  Coordinate, collect, analyze, and organize data across programs

¨  Report results to the Institutional Assessment Committee and to the Dean of Academic Affairs (DAA)

¨  Verify that the approved mission and goals of the academic unit are incorporated into Tk20

¨  Generate different types of reports available in the system, as needed or requested

¨  Offer assessment training activities (to academic deans, chairpersons, and assessment leaders) across programs

¨  Provide support to academic deans, department chairpersons, program coordinators, office directors, and faculty members on assessment process, tools and Tk20 platform

 

Assessment leader level

Faculty Deans, Chairpersons, Office Directors and Program Assessment Coordinator
¨  Coordinate with faculty the assessment activities, data entry and report activities

¨  Collect data and share results with program faculty and administrative personnel

¨  Coordinate with the assessment monitor the data entry process on Tk20 platform

¨  Enter into Tk20 goals, objectives and competencies for the academic or service programs

¨  Design the curricular map with the faculty members of the academic program

¨  Design with faculty members, incorporate and edit in Tk20 the Assessment Plan for the academic program or office

¨  Recommend the appointment of assessment subcommittee members

 

Faculty members level

¨  Participate in departmental committee (and subcommittees), assessment meetings and decision making

¨  Conduct classroom assessments

¨  Incorporate instructional strategies for direct and indirect measurement methods that support improvement of student learning

¨  Choose, develop and revise instruments for program assessment

¨  Report at least annually the results of assessment, in order to share ideas and strategies with peers

¨  Engage and support institutional and accreditation efforts

¨  Participate in planning and conducting program assessment and work with peers to improve program outcomes

¨  Review the Assessment Plan in Tk20 and its metrics

¨  Incorporate into Tk20 assessment results requested by the assessment leader

¨  Define the metrics for success

Table 2. Academic Assessment Duties Organizational Structure by Level

If the assessment leader is a professor (faculty member), then the Chairperson (Director of School or Director of Department) must share with the coordinator the following duties:

  • Supervise and collect assessment and data reports
  • Provide leadership to faculty and assessment coordinators
  • Communicate findings to department or academic division
  • Close the loop: analyze data, revise recommendations, and take actions for improvements

Strategy # 2: Use a common language

To move forward in the assessment endeavor, it is necessary to adopt a common language.  Some terms may be confusing, mainly for those who are starting to work in the assessment field. It is important to choose and define the basic concepts necessary for the assessment process, leaving others for later on. A glossary may also be developed and available for all campus personnel directly involved in assessment. For example, at the beginning of BSNS assessment work, Graduate Competence Profile was rapidly used as a synonym for “competencies” or “student learning outcomes”. Although they may not mean the same, professors needed to move forward with what was available at the moment. Competencies refer to the knowledge, skills and attitudes that the student must demonstrate upon completion of their study program. There is a “direct link between the competencies required for the practice of the profession and the contents of the academic programs” (Guide, 2016). Moreover, course objectives laid out in the syllabi should align with student competencies.

Strategy # 3: Have in advance, and available, four key assessment elements

Faculty members are better prepared to accomplish tasks in assessment if prior to start the use of Tk20 they have developed and available in a common format: (1) program competencies, (2) a curricular map, (3) metrics, and (4) an assessment plan. A simple curricular map represents an alignment of different curricular components, such as, goals, program objectives, competencies and courses (Guide, 2016). It shows where program assessment activities will take place.

The BSNS simple curricular map served as a guide for assessment planning; the courses where the assessment of a specific competence will be measured are distributed in this map. As shown in Figure 1, each competence will be measured in at least one course.  As a result of continuous

Art 6 fig1

Figure 1. Simple Curricular Map in Tk20 format

collaboration, faculty members have developed working documents to facilitate collecting data, such as the Metrics on One Document (Table 3). The expected learning outcomes are defined as the percentage of student population enrolled in the program (metric %) that achieves the expected minimum percentage (assessment %) using the assessment instruments approved by faculty members.  Following this definition, an expected learning outcome for BSNS students should read: “75% of students will get at least 65% in the rubric used to score a question in the second exam (CHEM1111)”.

Program: BS in Natural Sciences Expected Learning Outcomes  

COURSE

COMPETENCE (SLO) METRIC

(%)

ASSESSMENT  (%)
1. Describe the basic concepts of the natural sciences and technology. 70 70

70

70

BIOL 1101

MATH 1511

MATH 1512

2. Demonstrate the processes related to the administration, analysis and interpretation of data. 70 65

75

75

CHEM 1111

CHEM 2212

PHYS 3002

3. Critically evaluate a scientific article of a primary source. 70 70 BIOL 1102
4. Analyze, synthesize and communicate concepts effectively from a multidisciplinary point of view. 70 65

65

65

BIOL 1102

CHEM 1111

PHYS 3001

5. Compose written works using scientific information. 70 70

65

75

BIOL 1103

BIOL 1101

PHYS 3002

6. Use basic scientific equipment properly. 70 70

65

70

BIOL 1103

PHYS 3001

PHYS 3002

7. Carry out statistical analyzes of experimental data and reach conclusions about these. 70 70

65

65

BIOL 1103

CHEM 1111

CHEM 2212

8. Recognize the impact of the natural sciences and technology by identifying their responsibilities, purposes and usefulness. 70 70

70

BIOL 1102

BIOL 1102

9. Manage processes and related data guided by ethical principles and a responsible vision of their implications in the field of natural sciences and technology. 70 70

65

BIOL 1103

CHEM 2212

Table 3. Metrics in One Document

Since competencies articulate with course content, establishing a precise timetable for assessing the students´ competencies allows professors to fulfill the programmatic assessment requirements and to incorporate data into Tk20. In order to establish an assessment plan that can be easily followed, several factors should be taken into consideration. The trimester academic calendar and the professor teaching workload are conditions that may pose restrictions to the assessment process activities, including training in Tk20. To cope with external factors that faculty does not control, a simple assessment plan is established in which the assessment cycle can be accomplished within a three year (nine trimester terms) period, starting in 2014 (Table 4).

 

 

 

Profile of the Competencies of Graduates

Year Calendar/ Trimester
2014-2015 2015-2016 2016-2017 2017-2018
2015-33 2016-13 2016-23 2016-33 2017-13 2017-23 2017-33 2018-13 2018-23
1. Describe the basics concepts of the Natural Sciences and Technology. X
2. Demonstrate the processes related to the administration, analysis and interpretation of data.  

X

3. Critically evaluate a scientific article of a primary source. X
4. Analyze, synthesize and communicate concepts effectively from a multidisciplinary point of view.  

X

5. Compose written works using scientific information. X
6. Use basic scientific equipment properly. X
7. Carry out statistical analyzes of experimental data and reach conclusions about these.  

X

8. Recognize the impact of the natural sciences and technology by identifying their responsibilities, purposes and usefulness.  

X

9. Manage processes and related data guided by ethical principles and a responsible vision of their implications in the field of natural sciences and technology X

Table 4. BS in Natural Sciences Simple Assessment Plan (2015-2018)

As stated in the assessment plan, gathering results or evidence about student learning at different points in time, is articulated to course offerings. While, one-time measures taken from one to three courses per competence may not address the full range of the competence, professors have some outcomes to determine whether students are achieving the expected outcome. To carry out the actions needed for closing the loop for each competence, a subcommittee is activated (Figure 2). Faculty members do not have to wait three years for changes to take place, if needed. Continued efforts are established and intensity of work is reduced if faculty addresses one competence per trimester.  As stated before and according to the plan, results from all nine competencies can be analyzed within a three-year period and action can be taken to close the loop.

Art 6 fig2

 Figure 2. Assessment Cycle for each competence (Adapted from Maki, P. L. (2010).
 

Strategy # 4: Be clear and set standards    

When setting standards, it is very helpful to work in teams. For instance, it may be necessary to delineate what is being asked for and what it is not: the target is academic program assessment and not classroom assessment. Although classroom assessment may be part of the whole assessment project and can be included in discussions, decisions related to program assessment should be taken by faculty members as a group, not individually. As a group and in consensus, analysis provided by the faculty is the base for the actions to be taken.

Academic assessment is not intended to be an experimental research activity. Although it is systematic, it may not have the accuracy expected for experimental research. For example, in this kind of activity, it is not necessary to establish cohorts. Decisions are based on discussions related to specific assessment results, together with the experiences shared among faculty members.

The assessment process allows faculty to discuss ways in which the program can improve students learning.  Further, higher education institutions are frequently called to demonstrate accountability (Ewell, 2009). Institutions must gather information to provide evidence that graduates demonstrate the knowledge, skills and attitudes described in their program. The assessment process provides professors the opportunity to openly discuss issues related to improving the student learning outcomes, which otherwise might be hidden. Through assessment, faculty members determine ways to inform curricular improvement and demonstrate student academic achievements, and move toward a different teaching engagement, from an individualized classroom assessment culture to a more collaborative academic assessment culture.

Strategy # 5: Share information 

It is essential to promote teamwork to develop an effective assessment process. The BSNS faculty members meet weekly to share and analyze assessment results, determine if they meet expected learning outcomes, make decisions for improvement, and report progress in the Tk20 implementation.  Technical pitfalls related to Tk20 as well as issues related to the program under study are also addressed.  It is crucial to convey clear instructions and avoid misinterpretations that may delay the process. Assessment Committee members or faculty members should show the rationale used for decisions taken and promote cooperation among peers.

Concluding Remarks

Academic program assessment may become a frustrating and time-consuming activity. In this article, authors have briefly explained five strategies to help assessment coordinators and other leaders overcome these and additional challenges. By adopting a “make it simple” approach and applying at least the five recommended strategies, the authors have initiated the use of Tk20 platform in the implementation of the BSNS program assessment plan. Furthermore, a sustainable and systematic culture of assessment has been raised. Specific pitfalls have been identified regarding issues such as multiple measures competences, and course selection for the curricular map.

Since 2013, MC organizational infrastructure has improved to further support assessment endeavors. As an example, the recruitment of an assessment monitor has been a very meaningful action taken. The assessment monitor is the day-by-day liaison for deans and faculty members and works closely with program assessment coordinators.  Further, revised duties at different organizational levels have provided a sense of a common commitment towards the achievement of student learning (Strategy 1).

Performing academic program assessment as a non-research activity does not reduce its relevance. Assessment can be defined as “the systematic collection of information about student learning, using the time, knowledge, expertise, and resources available, in order to inform decisions that affect student learning” (Walvoord, 2010). “A good assessment is one whose results are used to improve teaching and learning and inform planning and budgeting decisions” (Suskie, 2009). The authors agree that assessment has two main purposes: improvement students’ performance and support accountability. In addition, research in assessment, which was not the goal of this case study, may further support the results achieved and give rise to further study. Research results may provide new insights into the relationship between the complex and many factors embedded in the assessment cycle (Strategies 2 and 4).

The authors recognize two advantages in using Tk20 platform for assessment initiatives. First, data information collected and analyzed related to a specific program assessment is filed and available electronically (Strategy 3). Second, diverse instruments to measure students’ learning outcomes can easily be shared among different academic programs and academic units (Strategy 5).  Weekly meetings of the program assessment committee has been a crucial activity during this Tk20 implementation phase. Professors actively engaged during this phase have learned substantially more about how to assess student learning at the program level. Assessment is a dynamic process and, as stated by Suskie (2009), it is a work in progress. As professors learn more about the benefits of digital systems management, additional strategies will be identified to make the assessment enterprise more manageable.

Acknowledgements

We would like to thank Migdalia M. Texidor (MC Dean of Academic Affairs) and Carmen Oquendo (MC Dean of Students) for their feedback on the draft of this article. We also thank Rose M. Vincenty (MC Assessment Monitor) for her support and Tk20 training, and the faculty members for their commitment in the implementation of the Tk20 BSNS program assessment plan.

References

Ewell, P.T. (2009). Assessment, Accountability, and Improvement. Revisiting the Tension. National Institute for Learning Outcomes Assessment. November, 2009. Retrieved from http://www.learningoutcomesassessment.org/documents/PeterEwell_008.pdf

Guide to Curriculum Development. Inter American University of Puerto Rico. Vice Presidency of Academic and Student Affairs and Systemic Planning. Spanish Version. (February, 2016).

Maki, P. L. (2010). Assessing for Learning (2nd ed).Virginia: Stylus Publishing, LLC.

Monitoring Report to the Middle States Commission on Higher Education. Inter American University of Puerto Rico, Metropolitan Campus. (April 1, 2015). Retrieved from http://www.metro.inter.edu/servicios/rectoria_doc/docs/Monitoring-Report2015.pdf

Self-Study Report to the Middle States Commission on Higher Education. Inter American University of Puerto Rico, Metropolitan Campus. (January 2, 2013). Retrieved from http://www.metro.inter.edu/servicios/rectoria_doc/docs/Self-Study%20Standards%201-14_Feb%204%202013%20RV.pdf

Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.). San            Francisco, CA: Jossey-Bass

Walvoord, B.E. (2010). Assessment Clear and Simple. A Practical Guide for Institutions, Departments, and General Education. 2nd Edition. San Francisco: John Wiley & Sons, Inc.

Trackback from your site.

Leave a comment