The paper has a paragraph or two dedicated to the software used:
> All Cognitron tests were programmed in HTML5 with JavaScript by AH and WT. They were hosted on a custom server system (Cognitron) on the Amazon EC2 that can support diverse studies via custom websites. The server system was specifically developed to handle spikey acquisition profiles that are characteristic of main-stream media collaborative studies, fitting the number of server instances in an automated manner to rapid changes in demand. Here, maximum concurrent participants landing on the website information page was ~36,000, with this occurring at the point of the documentary airing on BBC2 in May.
Why wouldn't they also publish the source code? I took a couple of minutes to play with the test and I think I found some bugs. I would love to check myself. Isn't it quite important to audit the source code, given that it is the thing that collected the data?
It almost certainly is. And that's not entirely awful its a tool used by scientists to do work.
(it is inexcusable in 2021 in my personal opinion, but in many ways science is still in the dark ages wrt some of the tools being used, just as the
ML community is learning a lot about things like reproducability and falsafiability from scientists with experience here)
I just with the community would share code more to prevent common mistakes and repeated effort and so we can all learn.
Imagine a world with publically well documented code that helps people learn science rather than scientists having to teach coders what they want and codes having to teach scientists intracasies of languages and hardware choices.
> All Cognitron tests were programmed in HTML5 with JavaScript by AH and WT. They were hosted on a custom server system (Cognitron) on the Amazon EC2 that can support diverse studies via custom websites. The server system was specifically developed to handle spikey acquisition profiles that are characteristic of main-stream media collaborative studies, fitting the number of server instances in an automated manner to rapid changes in demand. Here, maximum concurrent participants landing on the website information page was ~36,000, with this occurring at the point of the documentary airing on BBC2 in May.
Why wouldn't they also publish the source code? I took a couple of minutes to play with the test and I think I found some bugs. I would love to check myself. Isn't it quite important to audit the source code, given that it is the thing that collected the data?