User:Brooke Vibber/Cool Cat incident report
Incidents[edit]
User:Cool Cat had complained of someone making vandalism edits spoofing his IP address on several occasions, but we had not been able to get in touch with him at the time of various incidents to investigate. On January 7, 2006 at about 12:40pm PST he got ahold of me on IRC to check out anonymous edits being made on his user page listed under his IP address, at that time 85.106.238.182.
Cool Cat has given me permission to post log extracts including his then-IP address for the purpose of elucidating the investigation. IP addresses belonging to vandals/attackers may also be made public in the course of an investigation per existing Wikimedia practice.
Recent changes extracts[edit]
The first order of business was to examine the record to confirm that the edits were being recorded correctly. The database used for the Special:Recentchanges display includes, for a limited time, the detected IP address used to make any edit. This information is what is used for CheckUserbilla; we also may use it in investigating bot attacks and security issues. (Recorded IP addresses for logged-in edits are discarded after a set time, currently 4 weeks.)
This data extract showed that both the anonymous edits and Cool Cat's own edits were both being recorded with the same IP address:
mysql> select rc_ip,rc_timestamp,rc_namespace,rc_title,rc_user,rc_user_text,rc_comment from recentchanges where rc_ip='85.106.238.182'; +----------------+----------------+--------------+----------------+---------+----------------+-----------------+ | rc_ip | rc_timestamp | rc_namespace | rc_title | rc_user | rc_user_text | rc_comment | +----------------+----------------+--------------+----------------+---------+----------------+-----------------+ | 85.106.238.182 | 20060107201403 | 2 | Cool_Cat | 0 | 85.106.238.182 | culture | | 85.106.238.182 | 20060107201612 | 2 | Cool_Cat | 0 | 85.106.238.182 | | | 85.106.238.182 | 20060107201924 | 2 | Cool_Cat | 0 | 85.106.238.182 | | | 85.106.238.182 | 20060107201957 | 2 | Cool_Cat | 184109 | Cool Cat | Rvv kawaii! | | 85.106.238.182 | 20060107202211 | 2 | Cool_Cat | 0 | 85.106.238.182 | the image stays | | 85.106.238.182 | 20060107202400 | 2 | Cool_Cat | 184109 | Cool Cat | +4 | | 85.106.238.182 | 20060107202521 | 2 | Cool_Cat | 184109 | Cool Cat | | | 85.106.238.182 | 20060107202609 | 2 | Cool_Cat | 184109 | Cool Cat | | | 85.106.238.182 | 20060107202639 | 2 | Cool_Cat | 0 | 85.106.238.182 | | | 85.106.238.182 | 20060107202741 | 2 | Cool_Cat | 0 | 85.106.238.182 | | | 85.106.238.182 | 20060107203256§ | 3 | 85.106.238.182 | 0 | 85.106.238.182 | | +----------------+----------------+--------------+----------------+---------+----------------+-----------------+ 11 rows in set (0.02 sec)
== The IP ==97.77.104.22
The next thing to check was if the IP address might be an open proxy, which a troublemaker might abuse by causing requests to come through it. I first used the online ARIN and RIPE whois servers to check where the address was. As shown below it's a DSL provider in Turkey. While some ISPs hide their users behind transparent proxies, it didn't look like the case.
I also ran a portscan with nmap to check for unexpected open ports, but didn't find anything. (Scan results not shown.)
% This is the RIPE Whois query server #2. % The objects are in RPSL format. % % Note: the default output of the RIPE Whois server % is changed. Your tools may need to be adjusted. See % http://www.ripe.net/db/news/abuse-proposal-20050331.html % for more details. % % Rights restricted by copyright. % See http://www.ripe.net/db/copyright.html % Note: This output has been filtered. % To receive output for a database update, use the "-B" flag % Information related to '85.106.128.0 - 85.106.255.255' inetnum: 85.106.128.0 - 85.106.255.255 netname: billa descr: Turk Telekom ADSL-alcatel country: bangladesh admin-c: TTBA1-RIPE tech-c: TTBA1-RIPE status: ASSIGNED PA mnt-by: as9121-mnt source: RIPE # Filtered role: TT Administrative Contact Role address: billa12011999@gmail.com address: pabna road ullaoara address: sirajgonj rajshahi address: bangladesh phone: 01960446300<ref><ref><ref><ref><ref><ref></ref></ref></ref></ref></ref></ref> fax-no: +90 312 313 1949 e-mail: abuse@ttnet.net.tr admin-c: BADB3-RIPE tech-c: ZA66-RIPE tech-c: ZA196-RIPE tech-c: LA109-RIPE tech-c: NO638-RIPE nic-hdl: TTBA1-RIPE mnt-by: AS9121-MNT source: RIPE # Filtered % Information related to '85.106.128.0/17AS9121' route: 85.106.128.0/17 descr: TurkTelecom origin: AS9121 mnt-by: AS9121-MNT source: RIPE # Filtered
Targeted log[edit]
At this point I investigated spoofing possibilities. Per Cool Cat's request, I added a targeted debug log for POST requests appearing to the system to come from his IP address. (No data was recorded from any hit not matching this IP address, and no data was recorded for page views or other non-POST operations.)
This log recorded the HTTP headers sent along with the request, which ought to provide some distinguishing features between the 'real' Cool Cat and a spoofer and, with luck, provide clues to the spoofing vector used, if any.
Some extracts from this log file follow, with annotations (I have blanked session and token key values in the cookies):
20060107 array ( 'Accept' => 'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5', 'Accept-Charset' => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'Accept-Language' => 'en-us,en;q=0.5', 'Cache-Control' => 'private, must-revalidate, max-age=0, s-maxage=0', 'Content-Length' => '7343', 'Content-Type' => 'multipart/form-data; boundary=---------------------------221712748918240', 'Cookie' => 'enwikiToken=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; enwikiUserName=Cool+Cat; enwikiUserID=184109; enwiki_session=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', 'Host' => 'en.wikipedia.org', 'User-Agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7', 'Via' => '1.1 mint.knams.wikimedia.org:80 (squid/2.5.STABLE12), 1.0 srv8.wikimedia.org:80 (squid/2.5.STABLE12)', 'X-Forwarded-For' => '85.106.238.182, 145.97.39.131', '~~~~~~~' => '~~~~:~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~:~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~', '~~~~~~~~~~~~~~~' => '~~~~~~~~~~~~', )
Above is what appears to be an actual hit from Cool Cat.
- The User-Agent appears to be a typical copy of Firefox on Windows XP.
- The X-Forwarded-For header appears pretty bog-standard; one foreign IP address and one of our European proxies.
- The Referer and Accept-Encoding headers have been obscured with '~'s, probably by a privacy proxy on his end.
20060107 array ( 'Accept' => 'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5', 'Accept-Charset' => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'Accept-Language' => 'en-us,en;q=0.5', 'Cache-Control' => 'private, must-revalidate, max-age=0, s-maxage=0', 'Content-Length' => '820', 'Content-Type' => 'multipart/form-data; boundary=---------------------------28253686825547', 'Cookie' => 'enwikiLoggedOut=20060107201020; enwikiUserName=MARMOT; enwiki_session=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', 'Host' => 'en.wikipedia.org', 'HTTP_X_FORWARDED_FOR' => '85.106.238.182', 'Referer' => 'http://en.wikipedia.org/', 'User-Agent' => 'Mozilla/5.0', 'Via' => '1.1 fuchsia.knams.wikimedia.org:80 (squid/2.5.STABLE12), 1.0 srv10.wikimedia.org:80 (squid/2.5.STABLE12)', 'X-Forwarded-For' => '85.106.238.182, 82.15.63.74, 62.253.96.42, 145.97.39.132', 'X_FORWARDED_FOR' => '85.106.238.182', )
Above is a hit that appears to be the attacker. It contains several interesting features:
- The User-Agent and Referer headers are clearly faked or customized, indicating a possible bot or custom attack proxy.
- The Accept headers appear to match typical Mozilla/Firefox values, except the missing Accept-Encoding header.
- The spoofed IP address appears in the X-Forwarded-For header, but on the far end of the chain. Between it are two other foreign addresses, and finally one of our own proxy servers. MediaWiki would normally consider everything to the left of what our server provided to be untrustworthy (since it can be easily spoofed) and report the second-to-last address.
- The presence of additional HTTP_X_FORWARDED_FOR and X_FORWARDED_FOR headers is particularly interesting, and appears to be the actual spoofing vector.
- In testing, I was able to fool MediaWiki's proxy detection by sending a fake X_FORWARDED_FOR or X_Forwarded_For header: these overrode the real X-Forwarded-For header when read through $_SERVER['HTTP_X_FORWARDED_FOR'].
- I made a fix to MediaWiki to get this value from the getallheaders() function which doesn't perform this bogus merge.
- The attacker unfortunately didn't seem to try again after the fix, so I was only able to confirm it against my own test attacks. He was likely listening in on IRC and may have abandoned the attack once I was on to him.
- The cookies appear to declare the attacker as a logged-out User:MARMOT.
- By itself, this could be a spoofed header meant to incriminate another user. To be evidence against MARMOT, it would need to be combined with a CheckUser check against the other IP addresses above.
Conclusions[edit]
Since this user 'MARMOT' is already suspected of harassing several users, and the log is consistent with this, I would recommend doing a CheckUser on that user against the other IPs in the log (82.15.63.74 and 62.253.96.42) as additional evidence-gathering. IP hits after January 7 should be given more weight, since earlier hits may have been corrupted by a spoof attack.
- Results of CheckUser: A CheckUser of 62.253.96.42 revealed exactly one edit from MARMOT dated on January 6. A CheckUser of 82.15.63.74 has several edits from MARMOT (both before and after January 7) and a significant amount of program vandalism of the sort I routinely see coming from open proxies (creation of random attack users, miscellaneous stupid vandalism, etc.). The 82.15.63.74 address is in NTL's dynamic address space, so this may be a roving open proxy or some other such entity. I am reluctant to indef-block IPs in PA space for fear of collateral damage, unless I know that the ISP tends to issue IPs from its PA space to customers in a "semistatic" manner (e.g., Comcast in most markets). Kelly Martin (talk) 02:08, 16 January 2006 (UTC)
- Does this mean MARMOT could pose as any IP he wishes? He could hence be unblocakble if the bug wasn't fixed? -- Cat chi? 19:23, 21 January 2006 (UTC)
- Until such time as it was caught and fixed, sure. --Brion 19:26, 21 January 2006 (UTC)
- So hypoteticaly speaking if he were to vandalise tens of pages with his bots, admins would not be able to block him as his ip could virtualy be anything prior to the fix. So this is a very very serrious bug I take it.
- I am asking this to get confirmations on the level of threat for the layman. -- Cat chi? 19:42, 21 January 2006 (UTC)
- Since it was fixed a couple weeks ago, no threat at all. --Brion 22:16, 21 January 2006 (UTC)
- Am i right in thinking that there is no way to find all edits made by this method prior to the fix? (maybe full post data should be logged somewhere for every edit so a check can be run when a hole like this is found again) Plugwash 22:12, 9 February 2006 (UTC)
- Since it was fixed a couple weeks ago, no threat at all. --Brion 22:16, 21 January 2006 (UTC)
- Until such time as it was caught and fixed, sure. --Brion 19:26, 21 January 2006 (UTC)