用PHP写的一个去重统计的程序
时间:2008-05-27 来源:yhniejun
小弟的工作有时候要进行大数据量的统计与去重,考虑到高级语言的易用性,所以用PHP实现了一个去重统计的程序,效果还不错,统计速度也还行,在此拿出来跟大家共享一下,交流下设计思想。
#!/usr/bin/php -q
?php
/***
*
* File : skip_repeat.php
* Date : 2008/05/23
* Author : Nie Jun
* E-Mail : [email protected]
* Version : 0.1.0
*
*
*/
$fp = fopen('./key_record.txt', 'r');
if ($fp) {
while (!feof($fp)) {
$buffer = fgets($fp, 20);
$buffer = trim($buffer);
if (empty($buffer))
continue;
$md5_str = md5($buffer);
if (empty($obj_array[$md5_str])) {
$obj_array[$md5_str] = new OBJ_SIG;
$obj_array[$md5_str]->sig_stat = 1;
$obj_array[$md5_str]->sig_count++;
$obj_array[$md5_str]->sig_code = $buffer;
} else {
$obj_array[$md5_str]->sig_count++;
}
}
}
foreach($obj_array as $val) {
$w_fp = fopen('./key_code.csv', 'a');
if ($w_fp) {
fputs($w_fp, "\"'$val->sig_code\",\"$val->sig_count\"");
fputs($w_fp, "\r\n");
}
fclose($w_fp);
}
//print_r($obj_array);
//
class OBJ_SIG
{
var $sig_stat;
var $sig_count;
var $sig_code;
}
?>
#!/usr/bin/php -q
?php
/***
*
* File : skip_repeat.php
* Date : 2008/05/23
* Author : Nie Jun
* E-Mail : [email protected]
* Version : 0.1.0
*
*
*/
$fp = fopen('./key_record.txt', 'r');
if ($fp) {
while (!feof($fp)) {
$buffer = fgets($fp, 20);
$buffer = trim($buffer);
if (empty($buffer))
continue;
$md5_str = md5($buffer);
if (empty($obj_array[$md5_str])) {
$obj_array[$md5_str] = new OBJ_SIG;
$obj_array[$md5_str]->sig_stat = 1;
$obj_array[$md5_str]->sig_count++;
$obj_array[$md5_str]->sig_code = $buffer;
} else {
$obj_array[$md5_str]->sig_count++;
}
}
}
foreach($obj_array as $val) {
$w_fp = fopen('./key_code.csv', 'a');
if ($w_fp) {
fputs($w_fp, "\"'$val->sig_code\",\"$val->sig_count\"");
fputs($w_fp, "\r\n");
}
fclose($w_fp);
}
//print_r($obj_array);
//
class OBJ_SIG
{
var $sig_stat;
var $sig_count;
var $sig_code;
}
?>
相关阅读 更多 +