# The Future Of Judging



## BillyRain (Nov 9, 2016)

After a discussion on FB, I had a vision of what technology we may begin to use in the future. After thinking about it, it may not actually be that far away. I came up with a concept which (with a decent enough developer) is totally achievable and would be really cool.

Each station has a cheap android tablet (You can get them for like ~£20-30 now, all it needs to do is run one app and would be a good long term investment as they will replace all printing/paper costs for scorecards). The tablets run a custom app which may look something like this:

(adjustments would need to be made for different/BLD events etc. Perhaps there would be a dropdown to select the event with different layouts for each one. The master tablet at the data table could actually be used to change the layout on every tablet when the event changes!)







It basically replaces the stopwatch as it has this facility in the top left. No more losing them, breaking, alarms during the final (UKC2016) etc.

It also links back through a wifi network to the data entry table. The process then becomes thus:

1. Runner takes cube to station and calls for competitor.
2. Competitors have a name sticker or lanyard with their competitor ID on it (No more scorecard printing!).
3. Judge enters competitor ID into the app and confirms the competitors name.
4. Judge uses app to time inspection. Stopwatch gives an audio warning at 8 and 12 seconds (More accurate than human calling + one less thing for judge to forget) and also automatically forces a +2 if over 15 seconds and a DNF if over 17.
5. Competitor solves the cube and the judge enters the time into the app, selects penalty if applicable.
6. Judge and competitor both use an attached stylus to sign in the relevant boxes.
7. Judge presses SUBMIT and all of the solve data is sent to the data entry table where it can be immediately reviewed and entered into cubecomps. (No more waiting for scorecards/misplacing them/indecipherable numbers).
8. If needed, the judge can tap the "Call Delegate" button which works just like a supermarket checkout system when they need to call a supervisor. The staff table would have a device which would make a sound and alert staff as to which station number has summoned them. Hell it could even tie into the delegate's cell phone and vibrate when they are called with a push notification giving them the station number. The possibilities are endless.

Daniel Sheppard even noticed that it would be possible to plug the stackmat timers into the tablets and have them interact with the app the same way that they can with Prisma for instance to eliminate the need to even enter the time manually. I'm pretty sure we could then use a headphone splitter adapter to split off the data line so that you can still use the display. After all the data is transmitted via audio signals.

I just wanted to get your thoughts on this. I know that some organisations won't be able to afford to purchase a tablet for every solving station.. but for those who can it would be an awesome solution and as mentioned, eventually a good investment.

Just something else that in the future could make competitions much more streamlined, which is always the aim!

Or just tell me it's dumb 

Billy x

EDIT: Another thought, if the timer is tied into the app, the stop/reset button is no longer needed. The app would detect when the stackmat is started and will therefore be perfectly accurate in calling a penalty for inspection time!!

EDIT: I found a small flaw, with no scorecard there is no way of matching the cube to the competitor. I'll have a think on that.


----------



## Robert-Y (Nov 9, 2016)

Also we need to find a way to determine the next scramble to apply to the competitor's puzzle.


----------



## BillyRain (Nov 9, 2016)

I think at the end of the day there has to be a slip of paper to accompany the cube. Maybe you could make some reusable robust number cards with just the competitor ID on it. At the start of the comp everyone is given their competitor ID. I know it kinda replaces names with numbers, which isn't very nice.. but not that bad. Then instead of placing their cube on their named scoresheet, it's placed on their number. 

The scramble table could also even have a tablet or two with "Enter ID" on the screen, you enter the ID and it just spits out the correct scramble to you. No more getting the wrong/same scramble because the master tablet works out which scramble they are on and just provides it to the scrambler.


----------



## Robert-Y (Nov 9, 2016)

What about something like this:






Scramblers scramble a cube and somehow fill in this table with "scrambled" for a competitor whose cube they have just scrambled. When a result an attempt is complete this table somehow updates with either "extra" or "complete". From here after the cube is returned, the scramblers refer to this table to determine the next scramble to apply.

EDIT: Hmm... but then what about competitors who do not make the cutoff?...


----------



## BillyRain (Nov 9, 2016)

Well something similar to that will be running on the back-end yeah. But for the user experience, the scramblers just have to enter the ID which is accompanying the cube and then it shows the correct scramble for that competitor.

They scramble and then press "done" and it goes back to "Enter ID" for the next cube.

Even if the judging tablet awarded an extra solve, the scramble tablet would know this and would give the extra scramble at that time. You don't even need to know what number solve it is, as it just knows. Although the scramble number would probably display anyway..

Another cool idea, tablets have cameras. Build into the scrambling app could be a scramble checker. Once you've applied the scramble you just hold it in front of the camera real quick and it gives you either a green tick or a red cross according to whether the scramble is correct!


----------



## Jaysammey777 (Nov 9, 2016)

One solution to printing names would be to have colored cube covers, i.e. With duck tape. Have about 16 different colors. Then each competitor can be assigned a color


----------



## BillyRain (Nov 9, 2016)

Jaysammey777 said:


> One solution to printing names would be to have colored cube covers, i.e. With duck tape. Have about 16 different colors. Then each competitor can be assigned a color



This wouldn't really work as some groups are larger than 16 in bigger competitions. Also that could get confusing.

I think just using the competitor ID number and keeping a marker with the number on it with the cube in the tub is ok.


----------



## biscuit (Nov 9, 2016)

The biggest problem I've encountered while thinking of a system like this (I've thought about it and come to fairly similar ideas on much of it.) is making sure the cube stays tied to a person. You could have a slip of paper, but that feels like it could get lost fairly easily. What if we sere to make some way to attach an index card to the cube cover, which stays on that cube cover until the end of the heat? With cardboard ones, you could cut four slits like in a photo album. These index cards would also be how the puzzles are submitted. The process would go like this.

Organizers put out index cards with a name and competitor ID on it.

Competitors submit their puzzles by placing it on the index card with their name on it.

Organizers come along and attach the index cards to the covers, puts the puzzles in the covers, and take them to be scrambled.

A judge comes, picks up the box with scrambled cube and calls the name on the index card. 

The competitor solves the cube, signs etc. The judge places the puzzle back in the box, and is taken back to the scrambling table.

Scrambler inputs the competitor ID, gets the scramble, scrambles, it gets solved again etc.



What if the code breaks somehow? I feel like you'd still have to print out score cards just in case something goes wrong. The index cards could be replaced with score cards, so if the system breaks you just switch to using those.

Also, what if we have a time that is definitely not right? Data entry people make mistakes, so what's to say judges won't? We then don't have a score card to then check and find out what the actual time was. If we attach a stackmat then we're good, but with manual entry, judge mistakes become a huge deal

One advantage we have here, is we could assign a judge name to each table, so that in case of disputes (which should be dramatically decreased without ambiguous numbers etc) we know exactly which two people were involved and we can seek them out. 

I'd love to see it in practice, but we have some stuff to work out.


----------



## BillyRain (Nov 9, 2016)

biscuit said:


> The biggest problem I've encountered while thinking of a system like this (I've thought about it and come to fairly similar ideas on much of it.) is making sure the cube stays tied to a person. You could have a slip of paper, but that feels like it could get lost fairly easily. What if we sere to make some way to attach an index card to the cube cover, which stays on that cube cover until the end of the heat? With cardboard ones, you could cut four slits like in a photo album. These index cards would also be how the puzzles are submitted.



Yeah, basically what I was trying to describe when I said markers. Something more robust than a scorecard has less chance of getting lost.



biscuit said:


> What if the code breaks somehow? I feel like you'd still have to print out score cards just in case something goes wrong. The index cards could be replaced with score cards, so if the system breaks you just switch to using those.



Well as with everything it will be subject to failure, just like the timers themselves are. It would need to be thoroughly trialled before use. If the apps are stable enough we shouldn't have any issues. 



biscuit said:


> Also, what if we have a time that is definitely not right? Data entry people make mistakes, so what's to say judges won't? We then don't have a score card to then check and find out what the actual time was. If we attach a stackmat then we're good, but with manual entry, judge mistakes become a huge deal



Hmm, but how is tapping numbers on a keypad any different than writing on a scorecard. The same margin for human error is still there. 

I'm definitely now aiming for timers to be tied into the app, so this shouldn't be an issue anyway. 

Thanks for the feedback/suggestions!


----------



## biscuit (Nov 9, 2016)

BillyRain said:


> Hmm, but how is tapping numbers on a keypad any different than writing on a scorecard. The same margin for human error is still there.
> 
> I'm definitely now aiming for timers to be tied into the app, so this shouldn't be an issue anyway.
> 
> Thanks for the feedback/suggestions!



It's a lot easier to slip and hit the wrong key than write the wrong number. The time should be checked by both the competitor and judge, but as we know that doesn't really happen.


----------



## Lucas Garron (Nov 9, 2016)

I like the idea of trying to do this.
My main advice: Simplify, simplify, simplify.


----------



## Sajwo (Nov 9, 2016)

been there, done that


----------



## AlexMaass (Nov 9, 2016)

Jeremy Fleischman in the past has been working on something like this, pretty much switching to electronic scorecards. I think it could be cool to try this out, seems likely to be better, but there's always the saying "if it ain't broke, don't fix it".

One additional benefit is stuff like judges being careless and writing stuff like 12.34+ or having trouble with the times being hard to tell if its a 4 or a 9, for example, since its all electronic we people doing data entry into cubecomps won't have to deal with this.



Sajwo said:


> been there, done that


sadly that project went dead : (


----------



## biscuit (Nov 9, 2016)

Sajwo said:


> been there, done that



Problem is that is you have to have a custom piece of hardware. An app significantly brings down the price and usability.


----------



## Matt11111 (Nov 9, 2016)

biscuit said:


> Problem is that is you have to have a custom piece of hardware. An app significantly brings down the price and usability.


Brings DOWN the usability?


----------



## biscuit (Nov 9, 2016)

Matt11111 said:


> Brings DOWN the usability?



Err, brings up the usability.


----------



## jfly (Nov 10, 2016)

I spent a lot of time talking my head off about this sort of stuff in 2014/2015. I burned lot of time looking into the smartphone/tablet integration. Definitely take a look through http://www.jflei.com/2014/08/21/dialup-stackmat/ before you spend too much time on the hardware integration.

I spent so much time focused on the stackmat integration that I never really paused and asked myself what server these station tablets are going to talk to. Cubecomps wasn't an option because it's closed source, which is where CCM (https://live.cubing.net/) came from. It turns out the server side of this is a *ton* of work.

Unfortunately, CCM and a hypothetical mobile app to talk to CCM have basically been on hold since I was invited to create the WCA Software Team. Everything we did is open sourced, and I am definitely looking for help on the project.


----------



## pglewis (Nov 10, 2016)

My $.02: browser based would be far more future proof and OS agnostic vs. a native app.


----------



## AlexMaass (Nov 10, 2016)

jfly said:


> I spent a lot of time talking my head off about this sort of stuff in 2014/2015. I burned lot of time looking into the smartphone/tablet integration. Definitely take a look through http://www.jflei.com/2014/08/21/dialup-stackmat/ before you spend too much time on the hardware integration.
> 
> I spent so much time focused on the stackmat integration that I never really paused and asked myself what server these station tablets are going to talk to. Cubecomps wasn't an option because it's closed source, which is where CCM (https://live.cubing.net/) came from. It turns out the server side of this is a *ton* of work.
> 
> Unfortunately, CCM and a hypothetical mobile app to talk to CCM have basically been on hold since I was invited to create the WCA Software Team. Everything we did is open sourced, and I am definitely looking for help on the project.


I'm sure you could use cubecomps, Luis would love that.


----------



## biscuit (Nov 10, 2016)

pglewis said:


> My $.02: browser based would be far more future proof and OS agnostic vs. a native app.



I'm going to be trying to make a mock up of some sort, but if anyone more qualified (which isn't that hard) wants to make one, you'd be better and probably faster than I.


----------



## pglewis (Nov 10, 2016)

biscuit said:


> I'm going to be trying to make a mock up of some sort, but if anyone more qualified (which isn't that hard) wants to make one, you'd be better and probably faster than I.



I'd be happy to volunteer to help out with a project like that but I'm tied up with work in the short term and unsure when things will clear up enough that I'd be able to commit time and focus to it. Feel free to check back with me in the future if there's still a need, time for side projects will come back around at some point. 

Lots of tech savvy people around, though, and my instincts say a basic HTML/Javascript implementation would not be difficult and would work in practically any environment. Native apps seem sexier to everyone but they generally bring the weight of platform dependency along with them. Maintaining parallel forks/branches has never been a fun game. I'll also parrot @Lucas Garron on simplifying as much as possible.


----------



## AlphaSheep (Nov 10, 2016)

pglewis said:


> Lots of tech savvy people around, though, and my instincts say a basic HTML/Javascript implementation would not be difficult and would work in practically any environment. Native apps seem sexier to everyone but they generally bring the weight of platform dependency along with them. Maintaining parallel forks/branches has never been a fun game.


There are always options like Cordova which can turn an HTML/Javascript app into native apps for several platforms. They can have slight performance issues but for a simple data entry app it should work great.


----------



## BillyRain (Nov 10, 2016)

Obviously integrating with cubecomps would be amazing... but my idea is just to have a separate local database at the competition which can then be manually entered into cubecomps by the data entry person. Just like reading from a scoresheet, only there is no need to wait for them to come in anymore and data entry can be done so much faster. It also still means that a staff member checks every time manually before it's submitted.


----------



## DGCubes (Nov 10, 2016)

biscuit said:


> I'm going to be trying to make a mock up of some sort, but if anyone more qualified (which isn't that hard) wants to make one, you'd be better and probably faster than I.



Already started. 

I'd like to see how a few people would go about it though, so don't let it discourage you. IDK if I'd be able to implement everything honestly, but we'll see how it goes.


----------



## BillyRain (Nov 10, 2016)

I can't offer much help in terms of programming, but in terms of concept, ideas and graphic design for GUI stuff I can help anyone who needs it. Just hit me up on FB.


----------



## biscuit (Nov 10, 2016)

BillyRain said:


> Obviously integrating with cubecomps would be amazing... but my idea is just to have a separate local database at the competition which can then be manually entered into cubecomps by the data entry person. Just like reading from a scoresheet, only there is no need to wait for them to come in anymore and data entry can be done so much faster. It also still means that a staff member checks every time manually before it's submitted.



Oh okay... Maybe it goes live, but it has to be confirmed by someone. That way you do have truly live results.


----------



## Ollie (Nov 10, 2016)

An app that simply "reads" a completed scorecard (with the name, ID and times), interprets the times and pushes the data into Cubecomps via an API would be good.

The app would simply be:

An inspection time stopwatch (like Billy's original suggestion)
Could contain a link to the WCA regulations.
A data validation tool - it could contain Cubecomps style functionality that detects records and checks with the user.
A button to send a notification to a nearby delegate if there is a contention (like Billy's original suggestion) after they sign in to a competition with the app (applications here for registration too)

The main obstacle is messy handwriting. But I think handwritten digit recognition is probably pretty well-developed by now with machine learning etc. There is probably a free tool out there like TensorFlow that can be used too.

(Developers, go!)


----------



## Goosly (Nov 10, 2016)

Ollie said:


> An app that simply "reads" a completed scorecard (with the name, ID and times), interprets the times and pushes the data into Cubecomps via an API would be good.



Why does it need an inspection timer? An app for reading scorecards seems a seperate idea / tool, and you would only need one device with that app per comp.


----------



## Roman (Nov 10, 2016)

Ollie said:


> The main obstacle is messy handwriting. But I think handwritten digit recognition is probably pretty well-developed by now with machine learning etc. There is probably a free tool out there like TensorFlow that can be used too.



or we can just print scorecards with digit stencils and force judges to enter times properely


----------



## mark49152 (Nov 10, 2016)

1. Competitor sits at station. 
2. Face recognition software identifies them.
3. Competitor drops cube into scrambling bay, cover automatically slides over, and machine applies the correct scramble, after checking that the cube is solved and legal.
4. Competitor says "I'm ready" and speech recognition software causes cover to slide back.
5. 8 and 12 second warnings given automatically until timer started.
6. After the solve, cameras detect DNF or +2 scenarios and also verify that the cube is solved and was the same cube initially scrambled, by signature recognition of sticker chips or plastic texture.
7. Results are live on cubecomps within 0.01 seconds. 
8. That's enough of that, time for a cup of tea...


----------



## Ollie (Nov 10, 2016)

Goosly said:


> Why does it need an inspection timer? An app for reading scorecards seems a seperate idea / tool, and you would only need one device with that app per comp.



Doesn't _need_ one. But since we're discussing modernization of comps then an all-purpose 'WCA app' could include something to time inspection, automate and validate data entry, handle comp registration (by logging in with your WCA online account) or alert users of records in real time. 

Doesn't hurt to dream.


----------



## pglewis (Nov 10, 2016)

AlphaSheep said:


> There are always options like Cordova which can turn an HTML/Javascript app into native apps for several platforms. They can have slight performance issues but for a simple data entry app it should work great.



Yeah, that's why I qualified it with "generally". The important thing to weigh is what you would gain by making it an app. If you need access to device hardware then there may be no other option. I've spent too many years in legacy code-bases so everything added to a project simply means technical debt to me. For a quick little trip through how my mind works on this: every line of code and every dependency multiplies the vectors for breakage and will increase the long-term maintenance burden. Don't re-invent the wheel; definitely use well established and tested libraries rather than roll your own, but carefully weigh each and every one as if you were on a code diet. 

Making it an app it just so it's "an app" doesn't justify the overhead to me. Tools age, break compatibility, and die. It adds a learning curve for new contributors and even returning contributors that may have forgotten details. Cordova being open source is at least one positive for longevity/stability IMO. I haven't kept up with the state of the art on these "wrapper tools" so my past experience of them being dodgy and quirky colors my opinion as well.


----------



## Lucas Garron (Nov 10, 2016)

A web app can do everything you need this to do, especially on Android.

As an example, cubing.net/inspection is small, simple, performant, and cross-platform. If you "Add to Homescreen", it's just as good as an app.


----------



## BillyRain (Nov 11, 2016)

mark49152 said:


> 1. Competitor sits at station.
> 2. Face recognition software identifies them.
> 3. Competitor drops cube into scrambling bay, cover automatically slides over, and machine applies the correct scramble, after checking that the cube is solved and legal.
> 4. Competitor says "I'm ready" and speech recognition software causes cover to slide back.
> ...



LOL you've officially called it. We will come back to this in 20 years time and see what's gone down.


----------



## AlexMaass (Nov 11, 2016)

I bet if the WCA made an app, it would easily get taken down by Rubik xD


----------



## AlphaSheep (Nov 11, 2016)

pglewis said:


> Yeah, that's why I qualified it with "generally". The important thing to weigh is what you would gain by making it an app. If you need access to device hardware then there may be no other option. I've spent too many years in legacy code-bases so everything added to a project simply means technical debt to me. For a quick little trip through how my mind works on this: every line of code and every dependency multiplies the vectors for breakage and will increase the long-term maintenance burden. Don't re-invent the wheel; definitely use well established and tested libraries rather than roll your own, but carefully weigh each and every one as if you were on a code diet.
> 
> Making it an app it just so it's "an app" doesn't justify the overhead to me. Tools age, break compatibility, and die. It adds a learning curve for new contributors and even returning contributors that may have forgotten details. Cordova being open source is at least one positive for longevity/stability IMO. I haven't kept up with the state of the art on these "wrapper tools" so my past experience of them being dodgy and quirky colors my opinion as well.


I agree with pretty much everything you say here. To be honest I feel 90% of the apps out there would actually function better as web apps. You can even make an apps available offline these days, and as Lucas says, add it to your home screen and its just as good as an app.


----------

