h1

Are exams fit for purpose (part two) – what are the alternatives?

October 30, 2014

You dont fatten pigs 2

Last month’s blog came to the conclusion that examinations* are fit for purpose or at least “a purpose.”

They provide the student with a clear objective to which they can direct their efforts and focus attention and are a transferable measure of competency that can be assessed at scale. The “at scale” point is important as there are many ways of assessing competence but few that can cope with the need to test thousands of students all at the same time.

The main problem with examinations is that they don’t always examine what is most valued; the method of assessment often has significant limitations as to what it actually tests and the results are presented in league tables that give a far too simplistic view of success.

I am not sure we can resolve all of these but it might be worth exploring other options, specifically alternative methods of assessment. For example If you change the method of assessment from a formal, often timed written exam to say a portfolio of work, not only do you change the method of assessment but you will change what is being examined, two birds with one stone perhaps.

Different methods of assessing competence

Open book exams

Open book assessment offers a way of testing application rather than memory. Students have access to a text book that contains information relevant to what they are being asked. It’s the use of knowledge that is important, not the knowledge itself. The idea of open book could easily be adapted, why not allow students access to the internet during the exam, they could look up anything they wanted. Is this not more representative of what happens in the real world?

Take out exams

Similar to the above the so called “take out exam” allows the student to take the exam away to work on at home using whatever resources they prefer, books, internet etc. They return the next day with a completed answer. This can work better than you might at first think so long as you have a robust mechanism to detect plagiarism. There are several very good software packages that can spot the most sophisticated types of copying.

Case studies/simulations

A case study provides an environment for the student to demonstrate they can use their knowledge to solve problems and or offer advice in a virtual world. Most case studies tend to be written but this is one area that we could see some clever and affordable use of technology to better simulate the real world.

Performance tests

In a performance test students are required to demonstrate a skill/process, create a product etc while being observed by the assessor who will evaluate the performance. A great example of testing ability to apply knowledge but suffers from the subjectivity of the assessor and has limited application at scale.

Portfolios

Portfolios are most often collections of the student’s work that demonstrate their ability to perform a specific task. These can be simulations of the real world or portfolios of work actually undertaken on the job. A portfolio can include written documents, emails, audio or video recordings, in fact anything that provides evidence as required by the assessor.   Portfolios are perfect for assessing application but the process of assessment is expensive and not without bias.

Viva Vocal – (living voice) Oral exam

Often used to test PhD students, an oral exam gives the assessor chance to question the student. This is a very effective method where you are looking for higher level skill and depth of understanding. As identified last week it’s probably one of the oldest forms of assessment.

Digital badges – capturing the learning path

Being awarded a badge as recognition of achievement is something many will be familiar with especially if you were a boy scout or girl guide. But digital badging is new and becoming increasingly popular because of the internet. A good example would be linkedin and the badges awarded to you by others as recognition of certain skills.  Many of the assessment methods above provide a first past the post type of assessment, you pass and that’s it. Digital badging on the other hand is a form of lifelong assessment that evolves along with your career.

Digital badging for me is one of the most exiting forms of assessment and I am not alone Nasa have been using digital badging since 2011. Read more about digital badging.

Assessment in the future

Scanning for competenceThe list above is far from comprehensive and many other equally valid types of assessment exist e.g.  Role plays, Slide presentations, Assignments etc but what might assessment look like 15 years from now. Well how about using MRI scans to identify which parts of the brain are being used?  Not sure it will catch on but it would provide some interesting evidence as to how the students is getting to the answer, simple memory or a genuine and deep understanding .

*Examinations defined as a written test administered to assess someone’s level of understanding, knowledge or skills

h1

Are exams fit for purpose? (part one)

September 30, 2014

take-the-same-testI have written in the past about what passing an exam proves but have never questioned if exams achieve what they were originally designed to do, are they fit for purpose?

Firstly let me define what I mean by an exam. A written test administered to assess someone’s level of understanding, knowledge or skill that results in a qualification if successful. This is in contrast to a test which is a method of assessing someone’s level of understanding, knowledge or skill often as part of a course in order to provide feedback. A test does not have to be written. Although exams don’t have to be written either, many are and initially at least I would like to keep the definition as narrow as possible.

In order to answer the question, are exams fit for purpose we must first take a step back and look at how we got to where we are now.

 

A brief history of examinations

The first standardised test is believed to have been introduced by the Chinese in 606 AD to help select candidates for specific governmental positions. However most examinations around this time would have been oral, requiring the candidate to recite a dissertation or answer questions. Although there is evidence of written exams being used as early as 1560*, it was not until the 1820’s that many Universities began to adopt the practice. From 1850 onwards the written exam became the norm in most UK Universities. In 1854 under the Gladstone government selection of Civil servants was based on their ability to pass an exam, this time however it was written.

Bureaucracy – In 1917 to help bring some order to what had been described as chaotic the Certificate and the Higher School Certificate were introduced. Then in 1951 we had the General Certificate of Education (GCE) examinations, more commonly known as Ordinary ‘O’ level and Advanced ‘A’ level , these were normally taken at 16 and 18.

In the 1960’s the CSE (Certificate of Secondary Education) was born, opening up qualifications for all, not just those that went to Grammar school. However this two tier system was thought divisive and so in 1988 under the guidance of the then Education Secretary Sir Keith Joseph both sets of examinations were replaced by the GCSE. The GCSE was graded and contained credit for course work.  In 1991 the General National Vocational Qualifications, (GNVQS) were established intended to incorporate both academic and vocational elements, by 1995 these were accepted as ‘equivalent’ to GCSE.

In 2014 we find change again, gone is the course work and written examinations once again become the main method of assessment, although there will be grading, 1 to 9 with 9 being the higher mark. The exams will still be called GCSE’s, although officially they are known as GCSE (England). This is to avoid confusion with Wales and Northern Ireland, who are not changing.

Yes they are

Historically at least it would appear the purpose of the exam was to provide a recognised and transferable measure of competency in a given subject or discipline. The lack of transparency and consistency of the oral exam resulted in them being replaced with written ones and a more formal bureaucratic structure was developed to administer the process.

And in many ways there is very little wrong with this.

The problem is not with the exam itself, but with what is being examined. If as a society we value “thinking and creativity” for example, then should we not be examining these rather than subjects that require the candidate to do little more than rote learn facts.  Perhaps we should explore different methods of assessment, the written exam has its uses but hand written papers are looking increasingly outdated in a world that communicates electronically not only in short texts and tweets but with video and photos . In addition the way exam results are used in league tables to show winners and losers is divisive. It looks like a measure but has in fact become a target that schools and teachers must hit or be considered failures.

Please watch this it’s very funny…..and thought provoking

Not on the test

 

 

 

 

 

In the second blog about exams I want to look more closely at some of these points, in particular what other ways we can assess what people know.

*Assessment around this time was through debate between a number of learned people all at the same time and lasting for two hours or more.

h1

Big fish – little pond

August 31, 2014

Best be a Big fish in a Small pond

It’s taken me a little time to get round to reading the latest Malcolm Gladwell (MG) book, David and Goliath, underdogs, misfits and the art of battling giants. Although consisting of three separate sections they all examine the idea of what it means to have an advantage and how we account for the success of the underdog.

Of all the ideas MG lays before the reader, the one I felt was of most interest is something called the big fish little pond effect (BFLPE) and the theory of relative deprivation.

Relative deprivation theory (RDT)

Relative deprivation refers to the discontent people feel when they compare their positions to others and realise they have less. e.g. I judge BMW ownermyself to be successful on the basis I have a brand new car that cost £15,000.  That is until my neighbour pulls on the drive with a brand new BMW costing £30,000, now how successful do I feel?

MG applies this theory to the world of academia. If you take Harvard’s Economics PhD programme and consider the number of times each PhD graduate was published in the last 6 years, Harvard’s top students will do this 4.31 times. Those that are about 5th or 6th in the class publish .71 times and those that are about average .07 times. If however you compare these results to a “mediocre” school, say the University of Toronto, where MG went,  the top students will publish 3.13 times, those that are 5th or 6th .29 times and those that are average .05 times. The point being that students who attend a much lesser university but where they are top of their group perform considerably better than the 5/6th best at Harvard. The question is why?

The smarter your peer group the dumber you feel…..

This is where RDT comes in, we tend to judge our ability by comparing with others, and if you are in a class with very smart people who always do better than you, your perception of your own ability will be effected. The second problem is that this self perception will have a significant impact on your behaviour and ultimately what you achieve, hence the results above. The implication, you will achieve more if you are in a class with others of equal or less ability than yourself.

Bottom line, your performance will improve if you are a big fish in a small pond. It’s even called, the big fish little pond effect (BFLPE)

But what to do?

Admittedly you can’t always pick and choose your peer group, but you can be aware that comparing yourself with the very best may be having a detrimental impact on your own performance, so stop doing it! Instead be inspired by the best but compare your performance with those that are the same as you. Better still compare your current performance with what YOU have achieved in the past and if you are doing better you must be improving…..

David-and-Goliath-Malcolm-Gladwell

 

Listen to MG talking about relative deprivation theory or if you prefer the Big Fish little pond theory….

 

Listen to MG being interviewed about the book

h1

Sleep is for wimps – oh and successful students

July 31, 2014

Get some sleepAlthough I am sure someone is preparing for an exam this very minute, July/August are the traditional months to take a holiday and get some well earned rest. A holiday can be exactly what you need especially if you have just come to the end of a long period of study followed by in some instances, weeks of exams.

I have to express a personal bias in so much that I believe holidays are essential if you are to be at your best. For me this years holiday has to provide some degree of relaxation after what has been a particularly busy 6 months. I am looking forward to a change of scene, meeting different people and the freedom to wake naturally, feeling rested after a good nights sleep. Holidays are of course very personal and for some an adventure holiday, travelling to new places every day, might be far more desirable.

But one thing that all holidays should provide is the ability to relax and catch up on sleep, even if that means you climb two mountains, swim for three hours before crashing out in a state of satisfied exhaustion on the evening.

Sleep is essential for learning 

Of course sleep is something you should do “properly” every day, it’s just that we don’t. Modern life steals that vital rest time, this is acutely the case when trying to balance both work and study. Studying is often undertaken on an evening and sometimes late into the night as you effectively try to do, too much in too little time.   We now sleep less than we did 50 years ago, it used to be around 8.5 hours, it’s now only 6.5. The sleep should also be of high quality, yet our sleep is interrupted by the lights of mobile phones, and sounds made when texts arrive late into the night. In order to sleep better it is a good idea to avoid light approximately 30 minutes before going to sleep, yet how many read in bed from iPads or equivalent with the bright light emitted from the screen telling your brain to stay awake.

Why sleep is important

We have known that sleep has been important for many years but we didn’t know why, cognitive scientists now have some of the answers. There are three views as to why sleep is beneficial:

One restoration – some of our genes only turn on when we sleep, their role being to make essential repairs.

Two conservation – we sleep to conserve energy, and

Three consolidation – our brain revisit events and experiences, and begins to make sense of them, moving data into long term memory and solving complex problems.

Susanne Diekelmann at the University of Tubingen in Germany says “sleep helps stabilise the memories and integrate them into a network of long-term memory, it also helps us to generalise what we’ve learnt, giving us the flexibility to apply the skills to new situations. So although you can’t soak up new material, you might instead be able to cement the facts or skills learned throughout the day.”  Bodies need rest – the brain needs sleep Sometimes you may find yourself having to push sleep to one side and in specific situations thats fine.

It’s when lack of sleep becomes the norm that problems arise, the result is greater stress, poor judgement and ineffective learning.   So now the exams are over, take a break, get some quality sleep and try and make a few simple adjustments in you life so that sleep takes more of a priority.

It is a common experience that a problem difficult at night is resolved in the morning after the committee of sleep has worked on it.

John Steinbeck  

Music to help you sleep and two video to watch but not just before you go to sleep

TED neuroscientist Russell foster  explains more about why we sleep  

Arianna Huffington talks about the importance of sleep

h1

Teaching to the test – Interesting research but the fat lady is still in good voice

June 30, 2014

Fat lady still singingThis week researchers from the University of East Anglia released some very interesting findings that resulted from testing 594 bio-science students in their first week of term at five universities.

The students selected would be considered by many more than competent in their subject, almost all had a grade A*, A or B in biology at A-level. Yet when they were given 50 minutes to answer 38 multiple choice questions on cells, genetics, biochemistry and physiology from their A level core syllabus, they only got 40% correct. The period of time between the students sitting their A levels and the test was three months.

Lead researcher for the study, Dr Harriet Jones, said: “What our research shows is that students are arriving at university with fantastic A-level grades, but having forgotten much of what they actually learned for their exams.”  She went on to say that the trend to teach to the test, to ensure good results for schools’ reputations, was the problem.

The schools are to blame then

The facts of the research are clear, students who had successfully passed a test, were unable to pass a similar test three months later. The conclusion reached is that the students did not understand (see my blog on understanding) their subject well enough and passed their A levels probably using little more than memory. And who is to blame, the schools of course, for teaching to the test. Why the school do this is worthy of further debate, but government pressure and the impact of league tables will certainly be in the mix.

But do employers not accuse Universities of delivering up similar ill prepared students. The test is different but from the employers perspective the result is the same. A University student who professes to know something but when tested “in the real world” doesn’t.

Does this mean that Universities are also teaching to the test!

It’s about the test etc

The problem is not in teaching to the test; the problem is with the test, the pass mark and possibly the marking. If the test was more Testing but for what!aligned to what the student needs to know/do at a fundamental level, the pass mark sufficiently high and the marker having some degree of autonomy to form judgements, then the results would probably be different. It could of course be that the exams are easier – Exam chief: ‘you don’t have to teach a lot’ for our tests.

The big criticism of teaching to the test is, it results in a narrowness of understanding, little in the way of depth and does not push students to think in abstract and creative ways. But if the test, which incidentally does not have to be in the exam hall or on paper/PC was able to “test” for these qualities then teaching towards it would perhaps be more acceptable.

Bottom line

Teaching to the test is unlikely to change, in fact given the popularity of league tables  in education just now it may well increase, but with more effective testing the results might be better students, happy Universities and even happier employers.

 

 

h1

Twas the night before ………..the exam – but what to do?

May 31, 2014
keep-calm-and-study-all-night-5

Well not exactly all night

For students May and June are the main exam months. Studying and learning can be enjoyable…. honestly, but the fun has to come to an end and it does, with the exam. It cannot be avoided and so is best embraced, treat the exam as a game and you the player. What you need to do is give yourself the very best chance of winning.

Become a professional exam taker, someone who follows a process of preparation, very much like a top sportsperson. This means you personally need to be in the best physical and mental shape and have a series of exercises that will get you match fit.

Below is your training regime from the night before the exam – good luck

The night before

You should by now have:

  • Read through and reduced your class/tuition notes down to approximately 10 pages (20 max) of revision notes, see March Blog on how to prepare notes. You may have some professionally produced revision notes, but it is still best to make your own.
  • Practiced past questions on the key examinable areas both under exam and non exam conditions.
  • Started the process of memorising the revision notes.

Be realistic – The key to the night before the exam is to be realistic. You don’t have much time, so don’t think you can cover everything. Let’s assume you have 3/4 hours, 6.00pm – 10.00pm maybe.

Put to one side the large folder that contains all your notes taken throughout the term/year, and concentrate only on the 10-20 page revision notes.

Focus and memorise – In the 3/4 hours that you have you want to get an overview of the subject and focus on the areas that need memorising. These should be the key examinable areas and are most likely to be standard formats, definitions, lists, formulas s not given in the exam etc.  Memorising should include some rewriting of notes, but very little, talking out loud, drawing pictures, writing out mnemonics etc. See my blogs on memory, in particular: Thanks for the memories  and To pass an exam do and exam.

Admin – make sure you have set to one side everything you will need the next day. This includes your exam entry documents, calculator, gum, mints etc. You don’t want to be thinking of these in the morning. And of course make sure you know exactly what time you need to leave to get to the exam with about 1 hour to spare.

Physical and mental preparation – Drink lots of water, avoid tea, coffee etc as you will need to get a good night’s sleep. Exercise is an incredibly effective method of reducing tension and stress. So you may want to build into your 4 hours, 30 minutes for a run or brisk walk. This could be at the half way point of your evening, combining a well earned break with the exercise maximises your time.

Getting sleep is important, so avoid reading your notes and then going straight to sleep. Pack you notes away, put them ready for the morning, then physically go into another room if possible or even outside, watch TV for 10 minutes, something trivial or read a book. You need to break the state of mind from that of studying, relaxation leads to sleep not stress.

And finally keep a positive attitude, think about what you know and are good at and not what you don’t know and are bad at. Keep telling yourself that you have done everything possible, and if you follow these steps you will have. Thinking you know nothing and should have done more will not help at this stage, it’s a pointless thought strategy and not what the professional exam taker does.

The morning before

Set your alarm sufficiently early to give you at least another hour of revision. You don’t need to get out of bed, just continue memorising your notes. This is now about little and often, short 10 minute intervals. Don’t worry about falling to sleep in the exam; the adrenalin won’t let you.

1 hour before

What you do after arriving at the exam centre/School etc  is personal. Some will prefer to sit on their own going over the revision notes; don’t bother taking your folder of course notes. This is still very much about short term memory. Others will prefer to talk, chatting about nothing, just to stop them worrying. Both are fine.

After the examExam post it!

Afterwards is also a little personal, most will go home, but some will want to talk through what was in the exam, looking perhaps for some conformation they have not made a complete mess of it. Most importantly, if you have another exam, go home, put your old revision notes to one side, forget everything and start on your next subject.

The American basket ball player Art Williams had a good saying that I will leave you with. I’m not telling you it is going to be easy — I’m telling you it’s going to be worth it”

And although personally I found exams difficult I have never regretted the hard work, it was for me worth it.

And something to watch

How to: Cram the night before a test and PASS

Or you could try this

This blog is for Beth – good luck xx

h1

Eureka – I Understand Understanding!

April 29, 2014

I Understand!If you understand the subject you are studying your chances of passing the exam must be good.

A simple and perhaps obvious statement but what does understand mean and what do you have to do to truly understand something? Of course understanding is a key part of passing but it is not enough on its own, you can understand something yet fail because you run out of time, misinterpret the question, thought you understood but didn’t! etc.

To understand

The dictionary defines to understand as, to know what someone or something means, to grasp the meaning, to be familiar with, make sense of etc. Understanding is clearly different to knowing, for example, you may know that gravity is a force that pulls objects to earth but that does not mean you understand what gravity is or how it works. Of course you need both knowledge and understanding, the one is no good without the other. Examiners try to test for understanding by asking questions that require you to compare, contrast, explain, interpret etc.

Understanding is not a Eureka moment, it has different levels. It might seem that there is a point where you didn’t understand and then suddenly you did, a Eureka moment. In reality what you have done is move closer to gaining a better and fuller understanding. Ask any lecturer or teacher, often they will tell you they never fully understood something until they had to teach it, they just thought they did.

Proving you understand – The 6 facets of understanding

Understanding by design, Wiggins and McTighe (1998) is one part of an instructional design process that provides a very helpful framework we can use to explore the depth of understanding and perhaps more importantly what you can do to develop a deeper understanding. Think of it as a hierarchy with the easiest one first, the greater you’re understanding the higher the number.

1. Explain, the classic exam question – Explain to someone what the concept/idea means and say why. Explaining out loud to yourself or making a recording can be just as effective.

2. Interpretation – Relate the concept/idea to your own experiences, tell a meaningful story. Try to add something personal into your explanation. To do this you will need to reflect on past events, whilst attempting to find parallels with the concept/idea.

3. Application – Use the concept/idea in a different context. The ability to apply knowledge in different contexts (transfer) is a key milestone in learning as well as understanding. It should result in you never being caught out by a difficult exam question. Understand to this level and it doesn’t matter what the examiner asks.

4. Perspective - Read around the concept/idea, get other people’s views, and see the big picture. If your struggling with understanding, read another text book or my favourite is to go onto you tube and watch a video. The internet is great for discovering alternative views.

5. Empathy – Try to get inside another person’s feelings about the concept/idea. This is difficult as it requires you to put aside your feelings about the concept/idea and accept that it is not the only way of thinking about it.

6. Self Knowledge – Ask questions about your understanding, ask what are the limits of your understanding, what are your prejudices, become aware of what you don’t understand. Often called metacognition, the ability to think about thinking.

The Eureka moment

Understanding, like Eureka moments are not of course the result of sitting in a bath and suddenly finding you understand something you had previously found confusing. It is the gift of hard work and long hours of study, hopefully by trying some of the techniques above your depth of understanding will only improve.

Ps apparently the jeweller was trying to cheat the king….

Understanding by Design

Want to know more about understanding by design, watch this. 

 

 

Follow

Get every new post delivered to your Inbox.

Join 175 other followers

%d bloggers like this: