A while ago we've been reported that some users received worrying survey responses, with no text feedback and usually in the lowest range of the rating scale, although the contacts who allegedly submitted the ratings didn't even open the email surveys.
Explaining the problem
Our product team investigated the issue and concluded that some of the contacts that have received the survey had antivirus systems that checked each new email for unsafe/phishing URLs. These systems were randomly accessing URLs in the email surveys and checking for harmful content.
It’s important to keep in mind that the rating buttons in a survey are actually regular links, therefore, antivirus systems will automatically access some of the rating links.
In Retently, once a rating button is clicked in the survey, the rating is immediately added to the user’s dashboard, even if the survey respondent didn't hit the “Submit” response button. This is known as a “first intent rating”, which can be changed by the respondent before hitting the Submit button.
Following this logic, once an antivirus system checked a link in the email survey, the rating was automatically added to the user’s dashboard. Moreover, since the rating was already submitted, our system has considered the survey as “responded”, and when a real person accessed the survey, then only saw a “thank you” page, which added to the confusion.
Our solutions
1. Algorithm to avoid antivirus submitted survey responses
We've replicated this issue with several antivirus systems and implemented an algorithm to avoid it.
Our antivirus detection would automatically block the scores submitted by well-known systems such as Symantec, Microsoft Outlook Protection, Mimecast, MessageLabs, Barracuda Email Security Service, Trend Micro Email Security, and many more.
2. Algorithm to detect suspicious survey responses
However, these systems are evolving by adding new IP addresses to their network, and there are also other similar systems launched. To stay on track with these systems we’ve added another generic check in place which monitors antivirus-like patterns and logs them.
Once such a score, potentially left by a generic antivirus is detected, it will be displayed in your feedback widget with a warning ⚠ sign (as per the image below).
The antivirus warning will be added to the survey response with a 30 - 45 min delay, as it will take some time until our system will process the data.
If needed, on the Feedback page, you can also apply a filter to view all suspicious survey responses: Antivirus suspicion -> Is suspected.
3. CAPTCHA page
We’ve also added an intermediary CAPTCHA page that helps to stop antivirus systems’ access to the survey.
If the survey is accessed from an IP address that is part of a blacklisted network, the respondent will be asked to solve a CAPTCHA. Once solved, the respondent’s IP address will be automatically whitelisted, and they will no longer see the CAPTCHA page the next time they access another survey.
If needed, you can deactivate the CAPTCHA page individually per survey campaign. You will need to access a campaign's MISC section and switch the CAPTCHA option OFF.
What users can do
Even though Retently will do its part in preventing most of the antivirus submitted survey ratings, there still might be some that will bypass the filter and will end up in your account with the “antivirus warning” icon. At this point, it is up to you to decide whether to delete it at all or just ignore it from the Feedback page.
Note: When the respondent accesses the survey, our system will automatically identify and record their location based on their IP address. However, if an antivirus system was the first to access the survey, then its IP location will be recorded instead. Therefore, this might be another way to verify if the actual respondent or an antivirus bot provided a response.
Since the antivirus has already submitted a score to the initial survey, the original respondent would not be able to submit their feedback if they access the survey. Make sure to resend your email survey and give a chance to the actual respondent to leave their feedback.
We will further monitor our system to make sure our algorithms are continuously improved and such issues are avoided. If you have something to report on the received score, please get in touch with us.