Deep Learning Aftershock
Researchers from Google and Harvard published a groundbreaking paper on using DL to predict earthquakes' aftershocks in Nature last year. The paper was widely hyped but it was also criticized for using contaminated testing dataset.
The criticism earlier this year was rebuffed by the authors of the paper and the editors of Nature. Yet another blow was dealt to that state of the art paper now. Nature had to publish the new criticism this time.
The paper is paywalled in Nature but open access following the link below. I also included the link to the contamination criticism. I believe that the hype calls for more scrutiny and rigour of DL research rather than less.
One neuron is more informative than a deep neural network for aftershock pattern forecasting.
“DL should be used directly on observable and measurable variables avoiding or reducing de facto feature engineering and hidden assumptions. However, DeVries and colleagues used a combination of the stress tensor components as DNN input. The stress tensor is not measured but estimated on the basis of different assumptions (e.g. homogeneous medium, linearized elasticity theory), and measured or assumed quantities (e.g. distributions of rupture slip, Lamé constants, friction coefficient, regional stress), some of which are affected by large uncertainties. If not properly quantified, these uncertainties significantly influence local stress calculations, limiting the overall quality of any stress-based binary classifier (even for a complex DNN). In fact, accuracy seems to reach an AUC plateau of 0.85-0.86. A different and simpler approach for the same binary classification approach—is to use measurable variables, which are less affected by these uncertainties.
In the following, we show that a logistic regression based on mainshock average slip 𝐬 and minimum distance 𝐠 between space cells and mainshock rupture (i.e. the simplest of the possible models, with orthogonal features), provides comparable or better accuracy than a DNN. Both d and r [m] were obtained from the SRCMOD database.”
https://arxiv.org/abs/1904.01983
The first criticism:
Was this quake AI a little too artificial? Nature-published research accused of boosting accuracy by mixing training, testing data.
https://www.theregister.co.uk/2019/07/03/nature_study_earthquakes/
The notion of the superhuman performance of AI systems emerges from the comparison of AI system’s performance with human barebrain performance. Yet humans have been using computers and optimization algorithms for decades without any assistance of artificial neural networks.
“The fact that a neural network result can be closely approximated by a simpler model is a core result of our paper and one that we described in detail,” states the response of the team of Google and Harvard researchers who designed and trained a DNN with 13,451 paremeters only to predict earthquake aftershocks with equal or lower accuracy than a simple two parameter logistic regression model.
Human intelligence enhanced by a dumb machine in this case proved to be much more cost-effective, as well as time and energy efficient than a DNN based AI model. Is it just an exception? I don’t know but I am concerned. Along with other inefficqiencies training of an AI model with 13,451 parameters does leave a much more significant carbon footprint than applying a simple two parameter logistic regression model, doesn’t it?
My post on the scalability of AI from July 2017 still holds: “Is Machine Learning Ready to Scale?”
This post also provides some additional info on the hard problem of deep learning: “Superhuman Moron”
