Recently one of our team mate faced challenge in an anonymous SharePoint site, where we could successfully view/download all files except Excel Formats (.xls, .xlsx). We were not using Office web app.
 
Issue
While trying to open the file user sees the message "Operation in Progress", and then receive
"Excel Web Access  An error has occurred."

Or

"Access Denied"
 
Reason and resolution
 
We have resolved the issue by executing the following PowerShell commands so that it gives anonymous "OpenItems" permission rights to certain files that could potentially contain sensitive info with the following command:
·         $siteUrl = "http://yoursite";
·         $site = New-Object Microsoft.SharePoint.SPSite($siteurl);
·         $web = $site.OpenWeb();
·          
·         $enumPerms = [Microsoft.SharePoint.SPBasePermissions];
·          
·         Write-Host $web.AnonymousPermMask64;
·         $web.AnonymousPermMask64 = $web.AnonymousPermMask64 -bor $enumPerms::OpenItems
·         $web.Update();
·         Write-Host $web.AnonymousPermMask64;
·          
·         $web.Dispose();
$site.Dispose();
 For more details please refer