The tip of the iceberg – exam tipping is becoming obsolete

tip1

Assessment is changing, there was a time when all examinations were sat in a room, the answers would be hand written on a piece of paper and a retired English teacher would stand at the front reading out instructions as to what you could and couldn’t do in the next three hours.

Not any more…….you request a date that is convenient, turn up at the exam center, no longer is this a sports hall, it might be a driving school test center or the college you studied at. Then you log onto the PC and answer questions on the computer screen in front of you. The results may be immediate; it depends on if it is “human marked” or computer marked.

But in some ways these changes are only the tip of the iceberg!

What no past exam papers.

As examinations move into the digital world we are seeing other changes as well. There is a move towards objective testing, scaled scoring and examining bodies no longer providing past exam papers, what did you say, no past exam papers……!

This is partly down to the nature of the test i.e. you can’t provide an exact replica of a past exam question if it is an objective test. Remember objective test questions are randomly selected from a pool, and are different for each student. But there is also a shift towards some examining bodies only providing an example of the type of questions that could be set rather that a continuous flow of, the last exam papers.

If the test changes – how you study (and teach) has to change

Now for someone who has advocated that students analyse past exam questions in order to identify key areas so as to better direct their studies, this is a bit of a blow. It has also been the method I have used in the past to focus my own delivery in class and on line. Of course using past exam questions has always been much more than just spotting key areas, it is about focus, providing a place to start, showing content in the right context, helping with writing style etc.

There will still be past questions, sample questions will be provided. What we don’t know is how representative they will be of the examination. Or will it be as we have seen in the past with pilot and specimen papers, they change over time, drifting away from the original in terms of style and emphasis. Although I can see the logic in examining bodies not releasing papers, I hope they will continue to keep the sample papers fresh, in keeping with current thinking about the subject and how it will be examined.

What to do?

Students and tutors still need focus, there has to be emphases on key areas in order to chunk the content so that it can be more easily learned, it’s just that we won’t be able to use past questions or at least as much as we have in the past. That emphasis will now have to come from articles written by the examiners, examiner reports and syllabus weightings. If faced with a new subject where there is only one sample paper, it will be necessary to read the guidance from the body closely, noting reference to “this being a key part of the subject” or “one the examiner thought was answered badly in the past.” These together with the syllabus weightings and specific learning outcomes will have to be your guide. It is of course possible that the subject has not changed much from before and so some of the older past question can be used. As far as questions style is concerned then that will have to come from the questions and answers that are published, it may not be ideal but it’s the best we can do.

The overall impact of these changes is that students will have to know more, something that is hard to argue with. Students and tutors alike will have to devote far more time to the subject, which is fine if students have the time and can afford the extra costs involved in longer periods of study.

But it’s not all bad news, new technologies can help students make the most of dead time, studying on the train using their mobile phone for example. Also knowledge is more freely available than ever before as many top institutions provide a huge amount of free easy to access content online.

One final thought, examinations may change and they may not be fair but on the whole they are equal, everyone as before is in the same boat, and someone will always pass, wont they!

The future – Sitting the exam at home?

On line exams

An online student, all be it a mature one shows his ID to the online assessor

And maybe even the exam room will become obsolete. Proctur U is a US based company that also has a presence in the UK offering online invigilation. Watch this video to see how it works and judge for yourself

 

 

Video killed my teacher – metaphorically speaking

Video killed the radio star

What did you do the last time you needed to repair, cook or dare I say learn something? Did you google it and follow the link to YouTube? If so you are no different to the over one billion people who actively use YouTube every month.

This blog is not actually about YouTube but the medium of video and the increasingly important role it plays in our daily life and how we use it to learn.

 

 

Social learning and the bobo doll

Albert Bandura is the Professor Emeritus at Stanford University and is widely regarded as one of the greatest living psychologists. He is perhaps most famous for his Social learning theory, the theory of how people learn by observing others, and the bobo doll experiment. Click here to listen to Bandura himself explain.

Behavioral theories of learning suggested that all learning was the result of associations formed by conditioning, reinforcement, and even punishment, see Skinner and Pavlov. Bandura’s social learning theory proposed that learning can also occur simply by observing the actions of others. And that is where the true value of video lies, it is in the ability of people to watch what others do and learn from them.

What makes a good learning video?

Firstly, as with any form of delivery it needs to planned and structured. What is the purpose of the video, why use video and not send an email? Think about the audience, why will they want to watch it, what makes it relevant for them? Break it into three sections, a beginning, where you tell the audience what you are going to tell them, the middle, where you actually tell them, and the end where you tell them what you have told them.

Secondly It has to be relatively short, 10 minutes is a maximum. Even 6 minutes of good video takes a lot of planning, equally it wont test concentration levels too much. This does not mean you can’t record many hours of video, it just needs to be chunked, labeled and structured so it can be easily followed.

And lastly think about your delivery. Pace, tone of voice and body language all help the learner. This is where you manage the mood of your audience, if your happy they will be happy. Generally, speak more slowly than you would normally but be careful toooo slowww can be boring, vary how you say something depending on what you’re saying. Also think about the visuals and if it would be better to show an image rather than talk. But don’t go mad and put too much on screen all at the same time, it gets confusing.

Examples of good video 

But of course the best way to explain the power of video in learning is to show the videos.

1.The queen of cooking Delia is also the expert of slow deliberate, perfectly planned presenting. Here she explains how to cook an omelet, notice the attention to detail.  Ps Delia left school at 16 without a single GCE O-level. 3.43 minutes in length.

2.Here is someone who breaks the presenting rules, certainly the one that says don’t talk too fast. However, CGP Grey is great at using visuals, his dialogue is fast but incredibly informative, its packed with information, and it’s funny. If you are confused by the US elections, you won’t be after watching this.  5.19 minutes in length.

3.Crash course is a little like Khan academy which I have written before, what makes it different is the humour and how it is shot to camera using powerful visuals. Watch this clip if you want to learn about supply and demand. This pushes the boundaries time wise at 10.21 minutes.

4.This is the big one certainly as far as hits are concerned. James Stevens, Vsauceis watched by 19 million people. This one answers the questions as to, what would happen if everyone jumped at once? 7.12 minutes in length.

Why cramming works and making stuff up is okay

Will making stuff up

Will making stuff up

To a certain extent I have spent much of my career making things up. When I was a student that was not the case, I listened and learned and so when I spoke, I spoke with confidence that what I was saying was correct, because someone had just told me it was. Yet knowing is only the start, and in some ways a poor relative of the “figuring it out for yourself” technique.  I am reminded of quote from the film Good Will Hunting, which along with Dead Poets capture some really magical moments in learning.

Will Hunting – “See, the sad thing about a guy like you is in 50 years you’re gonna start doing some thinkin on your own and you’re gonna come up with the fact that there are two certainties in life.” “One, don’t do that.” “And Two, you dropped a hundred and fifty grand on a f***in education you coulda got for a dollah fifty in late charges at the public library.”

 Question practice – the secret to exam success

Having had no real formal educational training I have been exploring ideas as to why some techniques work and others don’t, why it is that student A passes yet student B who did exactly the same, failed. One clear observation from over twenty years in the high stakes exam world is that the most important activity that a student can engage in is, question practice. As a lecturer I would make statements, explain them using real world examples, get students to laugh, and maybe even enjoy the subject. But, the very best learning seemed to happen when the student was required to do a question. So it was with great interest that I read of some research that came out of the US in 2011, it’s called Retrieval Practice.

 Retrieval practice – the power of cramming

Retrieval practice is simply the process of retrieving something from memory.  So for example if I asked you, who was the Prime Minister that took us into the European Economic Community in 1973, you might say, on reflection Edward Heath. You already knew the answer but were forced to recall it. If however you were not sure who it was and were subsequently told (given feedback) it was Edward Heath and that Harold Wilson in 1975 held the first referendum, you are likely to remember both. But the most interesting and perhaps surprising aspect of this research is that not only can you recall the facts, it also leads to a deeper learning in so much that you can answer questions on related information. This in some ways gives credence to the idea that cramming information, maybe not at the last minute could be beneficial, not simply because you will remember it for a few hours’ but that it will lead to deeper learning.

Mark McDaniel is a psychology professor at Washington University in St. Louis

“We think of tests as a kind of dipstick that we insert into a student’s head, an indicator that tells us how high the level of knowledge has risen in there when in fact, every time a student calls up knowledge from memory, that memory changes.” “Its mental representation becomes stronger, more stable and more accessible.”

Jeffrey Karpicke, a professor of cognitive psychology at Purdue University

“Retrieving is the principal way learning happens.” “Recalling information we’ve already stored in memory is a more powerful learning event than storing that information in the first place,” he says. “Retrieval is ultimately the process that makes new memories stick.” “Not only does retrieval practice help students remember the specific information they retrieved, it also improves retention for related information that was not directly tested.”

Final thoughts

And so I am pleased to say that what I have observed in the classroom, that question practice improves exam results might be a little simplistic and that not only does it help students pass exams they might actually have been learning something at the same time:)

If you want to read more follow these links

To Really Learn, Quit Studying and Take a Test (New York Times)

Researchers Find That Frequent Tests Can Boost Learning (Scientific American)

For example – how to get higher marks in written questions

MORE-EXAMPLES

It’s great to be knowledgeable, but to pass an exam knowing the answer is often not enough. Questions set by examiners seek to do far more than identify people who “know stuff,” they want the student to prove understanding and that they can use the knowledge, not simply reproduce it.

The knowing doing gap

There is sometimes a disconnect between what you know and what you can explain. Have you ever said to yourself, “I know what I want to say but can’t find the words” or “what more can I say, I feel like I am just repeating the same point”. This may be the result of a lack of understanding and simply requires more study (see Eureka I understand understanding) or it might be that you just need a better way to think about what you’re trying to do.

Analyse, Explain – clarify – Example e.g.e.g.e.g.

Imagine you’re faced with a question, it asks that you, provide a possible  explanation as to why we have seen a fall in stock market prices in recent weeks and what impact this might have on  economic growth in the UK . Often the first problem is knowing where to start, below are a few ideas that might help.

You will need a few headings to help give structure, these can often be found in the question, here for example we could use, Why stock markets might fall and Impact on the UK. Then under each heading think about analysing, explaining, clarifying and giving examples. These are not headings; they are to help expand on what you have been asked to do and give a perspective from which to think.

  1. First you analyse – If you analyse something you break it up into smaller parts so as to gain a better understanding. For example going back to the question, perhaps we should identify exactly by how much the stock market has fallen, over what period, what other events were happening at the same time, do we have any theories that could help or theoretical models we could apply etc. By examining what you have found, something new and obvious may become clear.
  1. Then you explain – an explanation is an attempt to make clear what you mean. One way of doing this is by making a series of statements. So for example, if you noticed that during the period in which we had the fall in the stock market, China’s economy also slowed and oil prices fell to unprecedented levels. This might lead you to make the statement – one of the reasons for the fall in stock market prices would appear to be the slowdown in the Chinese economy and the fall in demand for oil.

A subset of explanation is clarification. Definitions are a great way to clarify exactly what something means and in what context it is being used. Here for example we might want to include a definition of economic growth.

  1. And finally the example itself, possibly one of the very best ways of explaining and a very powerful technique to demonstrate understanding.

Example “Metaphor’s forgotten sibling”. John Lyons

It may be a reference to a real world example. In the question we have to address the impact on the growth in the UK economy. If you gave an example of the last time oil prices were so low and what happened as a result you will not only be demonstrating breadth of knowledge but also moving the debate forward, suggesting perhaps that the same will happen again?

Real world examples demonstrate the complexity and unpredictability of real issues, and as such, can stimulate critical thinking.

Students learn by connecting new knowledge with their own prior knowledge and real-world experiences. Piaget et al

An example may also be a construct, something that you talk through to illustrate a point. For example, let us imagine the impact of falling oil prices on an engineering company in the West Midlands. A reduction in oil prices would result in lower transportation costs that could be passed onto customers in the form of lower prices, in turn this should increase demand.

“Examples are indispensable to the acquisition of knowledge and they appertain to the domain of intuition”. Kant

Although this blog has covered an approach to structuring written answers, it is the use of examples that for me is the most important. And if it was not obvious enough, look how many times I used examples to explain what I was trying to say …….

So you want to be an astronaut – assessment for astronauts

 

Astronaut-Tim-Peake-Dials-Wrong-Phone-Number-from-Space

Your there by yourself. There’s no doctor, there’s no computer engineer – so you have to learn all of these skills. Tim Peake

What is the purpose of assessment?

In order to gain a better understanding of the assessment process for astronauts, let’s ask a more basic question first, what is an assessment and what does it prove?

At one level it is simply a measurement of performance benchmarked against a given outcome or standard. The results can, help a teacher identify progress so they can adapt the next lesson, or give assurance an individual is capable of performing a particular task. This is probably the most important type of assessment for astronauts.   Modern high stakes “examinations however play another role; they offer the student a transferable badge of honour that can help open doors to better career prospects and increased salary. If you doubt the importance of this last point, click this link to read about the level of corruption in India and the lengths people will go in order to obtain a certificate saying they have passed an exam.

Employability gap – so what’s the problem?

However examinations on their own do not provide a guarantee that the individual who passes will be able to do the job.  And it is here where the disconnect becomes clear. Often the method of assessment is not based on what the student will be doing in the work place, and even if it is, it is not set in the same context. Many believe this gap has become ever wider as more and more students come out of school/university lacking the skills required by the employer.

But should the exam/assessment be changed to narrow the gap or is it the role of the employer to provide the necessary “on the job” education in the work place? However the problem with the employer being given the responsibility, it implies that what you study doesn’t matter, only that you do. This just seems wasteful, wasteful of time, money and effort. Assessment must get closer to measuring the skills required for one simple reason, what gets measured gets achieved, a cliché for sure but a true one.

Assessment for astronauts

Costs – The costs involved in training astronauts is a bit unclear, NASA will pay Russia $70m for a seat on Russian space craft in 2017 that includes training, figures of around $25m are also cited, Tim Peaks training was quoted as £16m so very similar.

Basic requirements – Mission Specialist (non pilots – most are pilots with a military background) include the following:

  1. Bachelor’s degree from an accredited institution in engineering, biological science, physical science, or mathematics. The degree must be followed by at least three years of related, progressively responsible, professional experience.
  2. Ability to pass a NASA space physical, which is similar to a military or civilian flight physical.
  3. Height between 58.5 and 76 inches.

Training – Tim Peak’s education started in 2008 when he was selected by ESA (European space agency) from 8,413 applicants. Tests at this stage included, intelligence, spatial awareness and concentration. Then for the 10 selected another 18 months of intense training followed, this time on topics as wide ranging as space law, rocket propulsion, spaceflight engineering and the hardest for Tim, learning to speak Russian. He was also subject to one week’s caving in Sardinia, Italy, with five other astronauts to simulate what it would be like in space, this was to build teamwork, problem solving and cope with poor hygiene facilities. Then it was one year of advanced training, including working underwater to simulate the lack of gravity. In 2013 it was time to go into space for the first time, this was the last test before being selected and then finally in 2015 Tim was allowed to do the job for real. In all it had taken 6/7 years.

What have we learned?

Clearly you need some underpinning academic skills i.e. a degree but a relevant one. The training is provided by the state, the argument being that it is for the greater good of society so the costs should be met by the tax payer. An interesting point for those that believe this should be the case for all education. But most importantly the assessments were all built around making sure that when Tim got into the space he could do the job. Very few of the tests required a piece of paper and a desk, and many were simulations of what he would meet in the real world.

Given the advances in technology the time is now right to introduce more simulations into the exam room, not so we can all become astronauts but to help prepare the next generation for the work place.

And just imagine your badge of honour when going for that next job – “what exams have you passed” “oh a few, but did I mention I was an astronaut…..”

 

27 Million People per day can’t be wrong – Gamification

League of LegendsThe statistics are astonishing, as of January 2014, over 67 million people play League of Legends per month, 27 million per day, and over 7.5 million concurrently during peak hours. And if your good at it the prize money for winning the world championship might get you to question your chosen profession, it was $2.3m in 2014 and 2015. Playing an on-line game is part of daily life for many people.

This blog is “of course” not about League of Legends. In fact I have to admit I had never heard of it, just shows how far out of touch you can become with popular culture. It’s not even about the gaming industry which is said to be worth £3.9 bn to the UK economy, it’s about a growing and fascinating area of learning called gamification.

Gamification is the use of game mechanics (rules, design and tools) in a non game context to better engage and motivate learners to achieve a desired objective. There are two types, structured, where you are looking to propel a learner through content and reward them for the desired behaviours and content driven where the game is the content i.e. the learner is a character in the game and is required to undertake tasks that are in turn rewarded.

Gamification techniques – Game mechanics

Games are not of course all the same but they do have similar characteristics, these “techniques” can then be used in a non game context i.e. a learning context. The idea being that if they engage and motivate the gamer, they will do the same for the student.  Games need some form of measurement to assess performance and a reward to act as an incentive.  Below is a note of some of the measurements and rewards used in gaming but could be adapted for learning.

  • Points – Used to keep score
  • Badges – visual stamps that are awarded to users on certain achievements and are normally displayed in their header and profile page
  • Levels – shows ranking and progress
  • Leader boards – a high score that is displayed for all to see
  • Rewards – not a badge but something tangible e.g. money….

Personal gamification

You don’t need to spend millions developing a game to get the benefits from gamification, and its not all about beating others, here are a few tips.

  • Set up a points system – identify the activities that will help you achieve your goal e.g. spend 2 hours each evening studying, 10 points. Answer 2 questions each evening, 20 points. Attempt the mock exam, 40 points. Score 50%, 80 points etc. Keep a running total of your points in a place that you can see when you study
  • Levels – Only move onto the next chapter or session when you have the desired points
  • Leader board – Keep a note of your highest score from the other subjects
  • Rewards – The best part. Set up a series of rewards e.g. a night off, go for a run, have a glass of wine, bar of chocolate etc. Increase the rewards as gaining the points becomes more difficult. If you beat your leader board score, then your rewards can be even greater, maybe a day out shopping/at the football etc. Why not ask others to contribute to the reward, if I get to the top of my leader board how about you buy me dinner. You will be surprised how many people, friends and family will effectively sponsor you.Other brands are available....

And finally tell your friends what you have done, “just eating a massive bar of chocolate which was my reward for scoring 80 points on my study game.”

Of course you might get fat doing this, but don’t worry there is another game that can help – it’s called weight watchers…..

Technology can help

As ever technology can help, check out this app HabitRPG – Click 

The mystery of scaled scoring

A mystery is defined as something that is difficult to understand and or explain. This describes exactly the situation we have at the moment as many examining bodies move their assessment towards scaled scoring and away from the more traditional percentage pass mark.  Some argue that scaled scoring is clearer in that it allows direct comparison of one examination to another but from my experience in the short run at least clarity is not the term many would use.

The raw score

How did I get on?For most students the way to find out how well they are doing is to have a piece of work marked. If the total marks available are 50 and the student scores 35 then as a percentage they have obtained 70% of the available marks. This is called the raw score. Interestingly it tells you very little, for example is 70% good or bad? For a raw score to have any meaning it needs to be compared with a standard. This could be with another student or group of students, and is an example of what is called norm referencing, now this clearly has meaning as you can tell if you are performing better or worse than your peer group. But the most common way in which exam results have historically been published is as a pass/fail against a pass mark or as it is sometimes called a cut score. If the pass mark is 60% and the student has scored 70% then they have passed.

The scaled score

However raw scores are only fair if the exam sat is the same for every student. It would be unfair to give each student a different exam and set the same pass mark, unless of course you believed the exam was of an equal standard.  This has been the argument in the past, yet producing an exam of equal standard is almost impossible, and this is why scaled scores are being used more often.

To use scaled scoring you need a range of marks, for our purposes let us assume the range or scale goes from 0 to 150. We then have to set a common passing score, say 100, this is the score on the scale that the student needs to achieve to pass the exam or test, and it will never change. What happens next is what makes a scaled score so useful.

ExpertsA group of experts including the examiner review the exam that has just been sat, they will probably mark a number of scripts themselves and will have access to the results from all other students. Then taking into account the performance of all students and their knowledge of this particular exam they will set the pass mark for this paper. Let’s assume that the pass mark is 32, if a student is awarded a raw score of 32 on this particular paper it will be equated to a scaled score of 100. All other scores above and below 32 will also be given a scaled score equivalent.

So on some levels using a scaled score is no different to the examiner setting a different pass mark for each exam, acknowledging that the level of difficulty will never be the same for any one examination. The reason the scaling is used however is because it can be stated that the passing score for the exam is always 100, this means the standard stays the same even though the marks may change.

So there you have it, no longer a mystery….I hope?

For more information and guidance watch the video on Pearson’s web site. Click on this link and scroll down.