The state of remote proctoring software is really, really bad.
Students are forced to install software that records video+audio of them, performs facial recognition, records their every keystroke for analysis, records their browsing history and open applications, aks them to show their entire room to the camera, keeps this data indefinitely with no privacy policy, and sometimes even gets compromised. Of course, sometimes it also doesn't even work, which makes it even worse!
There is generally no recourse for students who do not want to do this. Linux students are told to use Windows or fail the class. VM users are told they may be failed if they try to run the software in a VM (which is not allowed by the software and must be REed to attempt to begin with), and students that value their privacy or security (as this software is malware) have no recourse whatsoever often.
Regardless, droves of schools continue to spend absurd amounts of their endowment on expensive contracts with these remote proctoring malware companies, desperately attempting to hold up the facade that their degrees have a value that is somewhere near how much they charge in tuition.
I feel like I only escaped school recently myself, but it looks like it would be an even more miserable experience if I had to start the process again in 2020.
Disclaimer : one of my products is used for remote proctoring.
It depends on the vendor of the exam too, our proctoring sdk works on chrome and Firefox which means all major OSes too, we try hard not go outside browser context, we support android and iOS natively.
However many exam apps are very poorly written and have limited support for different OS, and usually never for mobile. I have seen full flash apps used today with 3months to deprecation.
we do face detection but not recognition and recommend using human live proctors for high stakes requirements.
The OS vendors are doing their part as well, we partner with both Apple and MS and give constant feedback, Apple announced AAC mode this WWDC and MS had number of APIs
Linux support is a factor in the education assements space due to chromebooks, most of my devs and half of my ops use linux , so I can say atleast one vendor cares.
Today there is profussion of vendors and overnight shops attempting to build in this space post covid-19, sales here is like any other enterprise product , tech merit is only one component, and nuances are lost on the customer in the hurry post covid 19, exam/certificate bodies have little scrutiny and rarely choose the better product for the user.
However in few years things will improve, buyers will understand the tech better, vendors will consolidate.
Hearing that one of the "vendors that cares" still does facial analysis gives me bad vibes.
Can you explain a bit about how the face detection works and what sort of detection data is sent outside the local client? If so, do you store or send it anywhere?
Make questions that you cannot easily cheat on. If I can simply google the answer and plug it in without changing anything, then it's not a good question. I think open ended problems are the way to go where only knowing can't cut it.
The solution is to not rely on remote proctoring at all. You ultimately cannot fully monitor an environment you don’t physically control, and in trying to do so, you’ll only end up pissing a lot of people off without doing anything to solve the problem. If potential cheating is such a critically important issue that proctors are ABSOLUTELY necessary, then don’t administer the tests online in the first place. Just augment the testing facility with COVID-19 related precautions, like required masks, hand sanitizer and spaced out desks.
Remote proctoring does not have to be automated. Plenty of tools do it with proctors monitoring the candidate.
You don't need it to be at location , you need human proctors .
Human proctors are expensive the bar exams are to blame here, if their question papers banks also too narrow so they have to do it the same time. Good exams will give you slots you can choose from , spreads the load.
No provider can come up with 30,000 proctors for just a day , even if bar exam boards were ready to pay, which they don't seem to be
I'm not talking about automated proctoring, I'm talking about remote proctoring. Although automated proctoring is its own can of worms, especially when it runs client-side and essentially demands that you cede control of your computer setup to it.
The sole purpose of having proctors is to alleviate cheating. Virtual proctoring is completely ineffective at doing so, and on top of that, effectively invites strangers to watch you in your own house and dictate what you can or cannot run on your own computer.
I've been closely watching this situation. It's a disaster. Basically, a bunch of places have been trying to do a remote bar exam since July, with various delays (Illinois, for example, is finally about to happen next week... theoretically). It's... just not working. Applicants are turning up security problem after security problem after security problem.
Which I think is predictable. The bar examiners (and many applicants) are incredibly paranoid about cheating, far more paranoid than normal university exams. So they're asking for, essentially, very strong managed device levels of security to prevent cheating, with stuff like facial recognition layered on top... and on an incredibly variety of applicants' own devices. It's like if the CIA had a BYOD policy.
I find the requirement of using a laptop without being allowed to use an external monitor to view potentially very long documents (half screen!) for hours on a tiny screen to be ridiculous.
Not just IL, also NY, CA, MA, FL, TX, PA, OH and a dozen more. ~35,000 nationwide, simultaneously!, next Monday and Tuesday, two sessions per day for two days. It all has to work perfectly 150,000 times!
The moral fitness test isn't designed to ensure that all lawyers are ethical. It only eliminates from consideration people whose past actions demonstrate that they have questionable ethics.
But many law students are young and have no paper trail/track record, so this isn't a very strong filter.
As an attorney who handles[] confidential client data running malware like this proctoring software is a breach of professional ethics, though not one yet acknowledged.
[] And not just will handle, because of limited cross state privileged many people who take the bar are already licensed and practising under another state.
I interviewed at an online proctor company, and I discovered that the build of chrome that they used tracked my eyes, tracked my mouse movements, and tracked my searches as I sat in their conference room and took some bullshit personality test.
I had a decent offer, but, still turned them down. Just wasn't interested in helping something like that succeed
The Respondus Lockdown browser, which I had to use last year, does something similar. I'm very pale and have white walls and literally had to hang a dark background because it couldn't see me, along with shining a bright light on my face.
Bar exams involve a significant element of speed and time management; being able to consult reference material on a secondary device doesn’t give you a huge advantage and may not be worth the risk of getting caught. Some bar exams are open book anyway. But being able to send a copy of the whole exam to someone else who can work on it in parallel, and send you completed answers to paste in, would give you a huge advantage. Locking down the only computer that can read the exam or submit answers creates a significant barrier to this.
>The target audience for such things probably only has one computer
...or they could borrow their friend's computer? Also, it doesn't have to be a computer. It could be a tablet/phone with usb-otg/bluetooth keyboard/mouse.
Seems like it's pretty easy to work around proctoring software. Some ideas:
* put the webcam in a separate room, pointing towards a body double and a clean working area. meanwhile, you're in the other room completes the test with full access to cheat materials by your side. you can also hire someone to do the test for you, and all you have to do is sit in the other room.
* plug a y-splitter into the microphone jack. in one end plug in a microphone, in the other plug in a phone playing a looped background noise track. the looped background noise track prevents the proctor from knowing that the microphone was unplugged. have someone that's out of the field of view of the webcam look at the monitor (with magnifying optics if necessary) and determine what materials need to be read out loud. When ready, disconnect the microphone from the y-jack and read the answers out loud. When done plug the microphone back in.
* have cheating materials on a secondary display (eg. monitor's PiP feature, or a tablet placed next to the monitor). get your accomplice to navigate to the materials as necessary, or use your feet to operate a mouse under the desk.
I could essentially do it in my system without any modifications.
I use desktop + laptop hooked up to the same screen and sharing mouse and keyboard through a usb switch. I can toggle between systems with a button on my desk without giving any clues of anything happening.
Completely different machine, so no hopes of ever seing that.
How much do people want to bet that this is using the haarcascade_frontface_default.xml classifier that's used in opencv face detection tutorials? I've seen similarly bad results on dark skin color.
My god, facial recognition that doesn’t accurately detect PoC? Who would have thought?
More seriously at this point why does any company shipped any kind of facial recognition software without verifying such catastrophic accuracy problems don’t occur?
In the past you could maybe argue that it was new enough to be a poorly understood problem, but for years now the most basic validation checks should include racial bias testing.
Students are forced to install software that records video+audio of them, performs facial recognition, records their every keystroke for analysis, records their browsing history and open applications, aks them to show their entire room to the camera, keeps this data indefinitely with no privacy policy, and sometimes even gets compromised. Of course, sometimes it also doesn't even work, which makes it even worse!
There is generally no recourse for students who do not want to do this. Linux students are told to use Windows or fail the class. VM users are told they may be failed if they try to run the software in a VM (which is not allowed by the software and must be REed to attempt to begin with), and students that value their privacy or security (as this software is malware) have no recourse whatsoever often.
Regardless, droves of schools continue to spend absurd amounts of their endowment on expensive contracts with these remote proctoring malware companies, desperately attempting to hold up the facade that their degrees have a value that is somewhere near how much they charge in tuition.
I feel like I only escaped school recently myself, but it looks like it would be an even more miserable experience if I had to start the process again in 2020.
reply