PHP网页采集的技巧

在采集网页数据的过程中,主要就是调用curl_url函数:

function curl_url($url,$if_echo,$follow_location,$use_ssl,$has_header,$is_post,$create_cookie,$load_cookie)
{
global $cookie_file,$post_data,$contents,$header;
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
if ($follow_location == "follow_location_yes")
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
if ($follow_location == "follow_location_no")
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_HEADER,1);
if ($use_ssl == "use_ssl_yes")
{
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 1);
}
if ($has_header == "has_header_yes")
{
curl_setopt($ch, CURLOPT_HTTPHEADER,$header);
}
if ($is_post == "is_post_get" or $is_post == "is_post_post")
{
if ($is_post == "is_post_post")
curl_setopt($ch, CURLOPT_POST, 1);
if ($is_post == "is_post_get")
curl_setopt($ch, CURLOPT_POST, 2);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
}
if ($create_cookie == "create_cookie_yes" or $load_cookie == "load_cookie_yes")
{
if ($create_cookie == "create_cookie_yes")
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
if ($load_cookie == "load_cookie_yes")
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
}
//curl_setopt ($ch, CURLOPT_PROXY,"http://127.0.0.1:8087"); //使用goagent代理
curl_setopt($ch, CURLOPT_TIMEOUT, 30); //超时设置
$contents = curl_exec($ch);
curl_close($ch);
if ($if_echo == "echo_yes")
echo $contents;
}

调用格式如下:

curl_url($url,"echo_yes","follow_location_yes","use_ssl_no","has_header_no","is_post_no","create_cookie_no","load_cookie_no");

在调用curl_url时,首先要确保echo_yes时能正常显示页面,如不能正常显示,主要注意的问题有:

[1]url地址,是否首尾有空格

$url = trim($url);
curl_url($url,"echo_yes","follow_location_yes","use_ssl_no","has_header_no","is_post_no","create_cookie_no","load_cookie_no");

[2]url地址是否发生跳转,follow_location_yes和follow_location_no这两种情况都要试试,在follow_location_no的情况后,看看转向后的地址是否正常展示;

curl_url($url,"echo_yes","follow_location_yes","use_ssl_no","has_header_no","is_post_no","create_cookie_no","load_cookie_no");
curl_url($url,"echo_yes","follow_location_no","use_ssl_no","has_header_no","is_post_no","create_cookie_no","load_cookie_no");

[3]url地址的协议,如果是https协议,要将use_ssl_no改成use_ssl_yes;

curl_url($url,"echo_yes","follow_location_yes","use_ssl_yes","has_header_no","is_post_no","create_cookie_no","load_cookie_no");

[4]url地址是否在goagent代理下能正常显示,如不能,需要关闭http代理功能;

//curl_setopt ($ch, CURLOPT_PROXY,"http://127.0.0.1:8087"); //使用goagent代理

在获取网页源代码后,通过正则获得需要的数据,这里主要用到的是preg_match和preg_match_all函数

问题1:匹配出字符串中所有的trackback地址

$content = <<http://pinklemonpark.com/shop_bb/book_appointment_2/211/90e/trackback
http://www.dear-one.net/zb/index.php?document_srl=2888&act=trackback&key=c74
http://www.medart.co.kr/xe/index.php?document_srl=1637&act=trackback&key=355
TrackBack URL

trackback:ping="http://blog.mayaline.com//u8uj/trackback.aspx?postID=373572"
EOF;
preg_match_all('{=['|"]([^()"']+trackback[^()"']*)['|"]}i',$content,$matches);
//此处 ['|"] 匹配单引号或者双引号
//此处 ([^()"']+trackback[^()"']*) 匹配网址,该字符串中包含trackback,且在trackback前面([^()"']+即1或多个字符且不包含引号和小括号,trackback后面([^()"']*即0或多个字符且不包含引号和小括号
for ($i=0;$in";
}

问题2:匹配出字符串中所有的name和value值

$content = <<










EOF;
if (preg_match_all('{]*name=["|'](.*?)["|'][^"']*value=["|'](.*?)["|']}is',$content,$matches2))
{
for ($i=0;$in";
}
}

问题3:匹配出符合条件的网址以及对应的日期

$content = <<

Acer Aspire V5-573PG BIOS 2.12 for Windows 8 64-bit

BIOS / Acer | 3.40 MB | Windows 8 64 bit

September 23rd, 02:21 GMT

NETGEAR WNR3500Lv1 Router Firmware 1.2.2.48 (NA)

Firmware / NETGEAR | 5.20 MB | OS Independent

September 22nd, 08:58 GMT

Tenda W311M WLAN Driver 3.0

Network Card / Tenda | 69.50 MB | Windows XP / Windows Vist…

September 21st, 23:55 GMT

EOF; //$today = date(“F jS”); //$yestoady = date(“F jS”,strtotime(“-1 day”)); //if (preg_match_all(‘{

]+>(.*?)s*(.*?)>(‘.$today.’|’.$yestoady.’)+([^><]+)GMT

}is’,$content,$matches2)) if (preg_match_all(‘{

]+>s*(.*?)>September (23rd|22nd)+([^><]+)GMT

}is’,$content,$matches2)) { for ($i=0;$i $title
$time
n EOF; } }

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>