Thread

After upgrading from 1.6.9 to 2.4.latest yesterday, I neglected to re-install the accessible captcha extension I was running in EE1.

That meant I was running standard EE captchas and with an hour of the upgrade, my oversight was exposed - I woke up today to about 150 spam comments which were clearly automated. I am usually only bothered by persistent manual spammers that number 1-2 a week. I don’t allow any links and parse out HTML the EE creates from BB code (be great to see that as a config option) so they don’t usually try more than once in one go.

That’s fine however - it was my oversight and I’ve now installed the EE2 version of the accessible catpcha and hope that does the trick.

The issue is that I was not email notified for any of them. When I woke up, I saw a single manual spam notification, logged in to remove it and saw all the automated ones. So somehow in addition to bypassing the standard captcha, they are also preventing email notification somehow. Notification is working as it always has done as evidenced by the one received for the manual spam, just not for these messages.

For the record, this is not (or doesn’t appear to be) related to the issue linked from my signature, because the spammer email addresses for each comment are not the notification emails for the weblog. Though for the record I’d love to see some form of communication from EE on that bug I linked to. Greg Aker has left, yet the bug is still assigned to him and it’s been a while….

I have extracted some logs that deal with the particular spammer and one of the pages affected to verify they were added on the front end rather than something nastier. I have masked the IP with “AT.TA.CK.ER”.

You can see the page is requested a few times using GET, including the captcha image (but not any other page assets), then a couple of POST’s which is presumably the posting of a comment, then a couple of regular GETs.

log:23062: AT.TA.CK.ER - - [16/Apr/2012:20:01:23 -0400] "GET /path/to/entry/ HTTP/1.1" 200 17347 "http://www.domain.com/path/to/entry/" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23062: AT.TA.CK.ER - - [16/Apr/2012:20:01:23 -0400] "GET /path/to/entry/ HTTP/1.1" 200 17347 "http://www.domain.com/path/to/entry/" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23064: AT.TA.CK.ER - - [16/Apr/2012:20:01:24 -0400] "GET /images/captchas/1334620883.47.jpg HTTP/1.1" 200 2589 "http://www.domain.com/path/to/entry/" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23067: AT.TA.CK.ER - - [16/Apr/2012:20:01:24 -0400] "POST /path/to/entry HTTP/1.1" 302 782 "http://www.domain.com/path/to/entry/" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23067: AT.TA.CK.ER - - [16/Apr/2012:20:01:24 -0400] "POST /path/to/entry HTTP/1.1" 302 782 "http://www.domain.com/path/to/entry/" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23078: AT.TA.CK.ER - - [16/Apr/2012:20:01:25 -0400] "GET /path/to/entry HTTP/1.1" 200 18209 "http://www.domain.com/path/to/entry" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23078: AT.TA.CK.ER - - [16/Apr/2012:20:01:25 -0400] "GET /path/to/entry HTTP/1.1" 200 18209 "http://www.domain.com/path/to/entry" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23082: AT.TA.CK.ER - - [16/Apr/2012:20:01:25 -0400] "GET /path/to/entry HTTP/1.1" 200 18206 "http://www.domain.com/path/to/entry" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23082: AT.TA.CK.ER - - [16/Apr/2012:20:01:25 -0400] "GET /path/to/entry HTTP/1.1" 200 18206 "http://www.domain.com/path/to/entry" "Mozilla/4.76 [en] (Windows NT 5.0; U)"

How might this spammer have suppressed notification?

I have secure forms turned on etc.

Also, when I click “Recent Comments” from the CP homepage, I get a list of sorted comments. However they are presented in ascending date order, despite the toggle being on descending. It means to see the most recent comment I have to scroll to the bottom, Switching the toggle presents comments from when the site started years ago…

Hi Daniel,

I’m sorry for the trouble this has caused you!

I take it that what you’ve shown here is a small sample? The timestamps on the requests suggest they were fired off rather quickly. I expect the sheer volume prevented the emails from being triggered. Either ExpressionEngine refused to keep up, or the server stopped them because of the rapid-fire requests.

Also, when I click “Recent Comments” from the CP homepage, I get a list of sorted comments. However they are presented in ascending date order, despite the toggle being on descending. It means to see the most recent comment I have to scroll to the bottom, Switching the toggle presents comments from when the site started years ago…

Yes, indeed. This is a known bug and has a fix available in the report.

I’ll ask the Devs for a clearer picture regarding the notifications.

Cheers,

Hi Dan,

Yes, that’s a small sample of a single instance of the post. I have 150 more.

Oddly, posting here has duplicated each line so here is a cleaned up version.

There is 2 seconds between the initial request for the page to load the crumb, the post and the subsequent redirect to the entry as is usual. Therefore it appears EE performs the post comment redirect, and I assume that should mean it’s stored the comment and executed the code to email the notification before it does so. If EE can redirect after the comment is stored but before the email is sent then I would be surprised but will wait to hear what the devs say. The server is more than capable of handling the load at the point this happened - the attack was distributed and I expect there were no more than 50 concurrent users with active sessions.

With 150 events over the course of time, it would seem odd not to get a single one.

log:23062: AT.TA.CK.ER - - [16/Apr/2012:20:01:23 -0400] "GET /path/to/entry/ HTTP/1.1" 200 17347 "http://www.domain.com/path/to/entry/" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23064: AT.TA.CK.ER - - [16/Apr/2012:20:01:24 -0400] "GET /images/captchas/1334620883.47.jpg HTTP/1.1" 200 2589 "http://www.domain.com/path/to/entry/" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23067: AT.TA.CK.ER - - [16/Apr/2012:20:01:24 -0400] "POST /path/to/entry HTTP/1.1" 302 782 "http://www.domain.com/path/to/entry/" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23078: AT.TA.CK.ER - - [16/Apr/2012:20:01:25 -0400] "GET /path/to/entry HTTP/1.1" 200 18209 "http://www.domain.com/path/to/entry" "Mozilla/4.76 [en] (Windows NT 5.0; U)"
log:23082: AT.TA.CK.ER - - [16/Apr/2012:20:01:25 -0400] "GET /path/to/entry HTTP/1.1" 200 18206 "http://www.domain.com/path/to/entry" "Mozilla/4.76 [en] (Windows NT 5.0; U)"

Yep- seems very odd.  And for the record- taking another look at that linked bug report.  Thanks for confirming the emails don’t match.

A couple of questions as I dig in:
1. How were you set to get the notifications?  Were you the author- and it was set to email the author; in the list of folks to notify of comments for that channel? 
2. Were all the comments to the same entry- or were multiple entries getting hit w/the spam comments that didn’t trigger notification?
3. You’re running 2.4.  Heh- not a question, but I wanted to ‘note to myself’.
4. Do you have comment moderation on??

Yep- EE shouldn’t be hitting a ‘too many comments’ threshold (if it is, it’s a bug)- it could be a server limit on how many in ‘x’ amount of time.  But even then- would seem odd none of the spam comments triggered a notification.

Thanks in advance for added info.

*Note- edited to add #4.

Hey Robin,

Thanks for stopping by.

1. I was set to get them both as the author and as the recipient of all notifications for the two channels affected. I have other channels seemingly not affected, but the two channels in question are right at the forefront in terms of percentage of total pages and and presentation for bot crawling so this is not surprising. I will check the other channels and do a diff to see if there are any clues vis a vis different settings, but I doubt it.
2. Multiple entries.
3. Yep -latest and greatest.
4. Comment moderation is off.

Lastly - nearly all of the 150 spam posts were from the same client, same IP.

Cheers

K- appreciate all of the data- I’m still not seeing how they did it.  So, a couple of more questions and a request:

1.  What extensions did you have installed at the time- and if possible, can you look in the db table exp_extensions and see if any are using the hook ‘insert_comment_insert_array’ or anything else that looks comment related.
2. Do you have a ‘comment timelock’ set?  That’s in ‘Admin- Channels’ in ‘Edit Channels’- the ‘Comment Re-submission Time Interval’?  And if so, what?  And if so- does it look like the comments violated that setting?
3. In ‘Admin- Security and Privacy- Security and Session Prefs’ do you have ‘Deny Duplicate Data?’ turned on?  And if so- were there duplicate comments?

So basically- I’m trying to spot if they managed to get around other security features and whether it’s possible an add-on is taking over some of the comment processing.

I’m also wondering if it’s possible for you to email me the relevant logs- edited down to the spam bits is fine.  And I totally understand if you can’t.  But if you can- you can zip it up and send it to: .(JavaScript must be enabled to view this email address).

Appreciate all the info- once I hear back I’m going to pull a dev in with me and go through everything and get this nailed down.  Right now, my best guess would be an add-on.  But that just seems unlikely, so….

1. Very few and nothing on the comment side - I use the standard comment module as vanilla as you can get. Low Seg2Cat & Matrix were installed at the time. In terms of Modules, nothing not first party that’s installed and enabled by default. Plugins - Low Replace, TruncHTML and SuperGeekery Tag Stripper.

Nothing appears to be using that hook.

2. Set to 10 seconds. Interestingly a cursory glance suggests they are thwarted by the timelock. They try a new POST on another entry a few seconds after the first one, then repeat the POST on the same entry until 10 seconds have passed. Sometimes they get ahead of themselves and try after after 2 seconds again, but the process repeats until they have success.

3. Deny duplicate is on, and sadly I do not have a backup of the comment data any more to check. They were all very similar I recall but probably with differences.

Do you just want logs in the same format as above (albeit the entire set)?

I can also switch off the Captcha that works in some kind of honeypot operation? I figure getting access to the access POST payloads may be key to resolving this so I will look into that.

Update to this - I am now working on the theory that these messages were marked as spam and delivery was prevented that way. My email is centralised in Gmail and there are no emails in the Spam folder, but this address in particular is forwarded through another service so I will check that. It’s not conclusive, but it’s my best theory to date.

Hi Daniel,

I haven’t ran this by Robin, but I did bring this to Kevin’s attention and our consensus is in line with yours.

The high frequency just overwhelmed either EE or the mail server with sheer volume.

Do let us know what you find out!

Cheers,

Just stopping in to say I got tripped up the other day by this exact issue- email I was looking for getting trapped by Gmail’s spam filter (I did find it in the spam folder, once I went to look there).

Because- yes.  We’re not spotting how it would have been subverted otherwise.  Do let us know once you’ve had a chance to check the other server.  I suspect you’re right- but confirmation would be of the good.

As it turns out I am unable to verify the receipt of those emails by my hosts email system which forwards the email, so given I am fixed with the working Captcha and the most recent theory seem to be the most likely, feel free to resolve.

Would still appreciate you taking a look at that “bug” referenced above Robin to see if that hole was fixed in EE2.

Thanks