HTML will always allow user to click 'All files'
Whether you restrict the "accept=" using the extension or the mime type or both, the user can always switch to 'All files'.
Javascript is needed
For example,
var selectedFile = document.getElementById('csvFileInput');
selectedFile.onchange = function(e){
switch ((this.value.match(/\.([^\.]+)$/i)[1]).toLowerCase()) {
case 'csv':
case 'txt': /* since they might have a CSV saved as TXT */
/* ... insert action for valid file extension */
default:
/* Here notify user that this file extension is not suitable */
this.value=''; /* ... and erase the file */
}
};
Server side validation is also needed
Remember the user can always defeat the process by renaming (for example) an .exe to a .csv, so where relevant you should also check the content of the file to meet your criteria, on the server side.