IDC头条提示词
封装一个现代的PHP CURL函数 要求足够健壮 1. 100% 模拟真实浏览器特征 2. 支持携带cookies请求 读取cookies 如果不存在就建立 不同网站使用不同cookie文件 3. 支持代理IP 代理IP池，代理IP读取 proxylist.txt 文件 每行一个 如果有多个，就随机抽取一个 4. 支持多种请求模式 POST GET 等等 5. 健壮的错误管理和记录 6. 支持Debug模式 调试请求和返回以及Header 7. 需要兼容 PHP 7.4 – 8.4
Modern PHP cURL wrapper for robust browser-like HTTP requests
Below is a self-contained, production-ready PHP helper that meets your requirements: realistic browser fingerprint, cookie jar per site, proxy pool from file, multiple HTTP methods, robust error handling, and optional debug. It is compatible with PHP 7.4–8.4.
Usage
<?php
require 'HttpClient.php';
$client = new HttpClient([
'cookie_dir' => __DIR__ . '/cookies', // where to store cookies per domain
'proxy_file' => __DIR__ . '/proxylist.txt',// optional: path to proxy list
'timeout' => 20, // request timeout (seconds)
'connect_timeout' => 10, // connect timeout (seconds)
'retry' => 2, // automatic retries on transient errors
'retry_delay' => 500, // ms between retries
'debug' => false, // enable debug prints
'log_file' => __DIR__ . '/httpclient.log'// error/log output
]);
// Example GET
$response = $client->request('https://api.ipify.org', [
'method' => 'GET',
'headers' => [
'Accept' => 'text/plain'
],
'proxy' => true // enable proxy pool (random pick from proxylist.txt)
]);
// Example POST with JSON
$response2 = $client->request('https://httpbin.org/post', [
'method' => 'POST',
'json' => ['hello' => 'world'],
'proxy' => false, // direct
]);
if ($response['ok']) {
echo "Status: {$response['status']}\n";
echo "IP: {$response['body']}\n";
} else {
echo "Error: {$response['error']['message']}\n";
}
Features
- Browser-like fingerprint: realistic modern headers and rotating User-Agent pool.
- Cookie management: per-domain cookie jar auto-created if missing; reads and persists cookies.
- Proxy pool: reads proxylist.txt (one per line); random selection if multiple; supports HTTP/SOCKS5.
- Methods: GET, POST (form and JSON), PUT, DELETE, PATCH; raw body and query string supported.
- Robust errors: categorized errors, retry logic, detailed log file records, timing metrics.
- Debug mode: dumps request line, headers, proxy, cookies path, response headers and body snippet.
- PHP 7.4–8.4 compatible: no new language features required.
Implementation
<?php
declare(strict_types=1);
class HttpClient
{
private $cookieDir;
private $proxyFile;
private $timeout;
private $connectTimeout;
private $retry;
private $retryDelayMs;
private $debug;
private $logFile;
// Modern desktop Chrome/Firefox UA pool (updateable)
private $userAgents = [
// Windows Chrome
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36',
// macOS Chrome
'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36',
// macOS Safari
'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15',
// Windows Firefox
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0',
];
public function __construct(array $config = [])
{
$this->cookieDir = $config['cookie_dir'] ?? __DIR__ . '/cookies';
$this->proxyFile = $config['proxy_file'] ?? null;
$this->timeout = $config['timeout'] ?? 20;
$this->connectTimeout = $config['connect_timeout'] ?? 10;
$this->retry = $config['retry'] ?? 2;
$this->retryDelayMs = $config['retry_delay'] ?? 500;
$this->debug = (bool)($config['debug'] ?? false);
$this->logFile = $config['log_file'] ?? __DIR__ . '/httpclient.log';
if (!is_dir($this->cookieDir)) {
@mkdir($this->cookieDir, 0775, true);
}
}
// Main request entry
public function request(string $url, array $options = []): array
{
$method = strtoupper($options['method'] ?? 'GET');
$headers = $options['headers'] ?? [];
$query = $options['query'] ?? [];
$form = $options['form'] ?? null; // application/x-www-form-urlencoded
$json = $options['json'] ?? null; // application/json
$body = $options['body'] ?? null; // raw body
$proxyEnabled = (bool)($options['proxy'] ?? false);
$proxyOverride = $options['proxy_string'] ?? null; // e.g. socks5://host:port
$cookieFile = $options['cookie_file'] ?? $this->cookiePathForUrl($url);
$followRedirect = (bool)($options['follow_redirects'] ?? true);
$maxRedirects = (int)($options['max_redirects'] ?? 5);
$ua = $this->pickUserAgent();
// Merge browser-like headers with user-provided headers
$defaultHeaders = $this->defaultHeaders($url, $ua);
$headers = $this->mergeHeaders($defaultHeaders, $headers);
// Build URL with query
if (!empty($query)) {
$qs = is_array($query) ? http_build_query($query) : (string)$query;
$url .= (strpos($url, '?') === false ? '?' : '&') . $qs;
}
$proxy = null;
if ($proxyOverride) {
$proxy = $proxyOverride;
} elseif ($proxyEnabled && $this->proxyFile && file_exists($this->proxyFile)) {
$proxy = $this->pickProxyFromFile($this->proxyFile);
}
$attempts = $this->retry + 1;
$lastError = null;
$response = null;
for ($try = 1; $try <= $attempts; $try++) {
$response = $this->execCurl($url, [
'method' => $method,
'headers' => $headers,
'form' => $form,
'json' => $json,
'body' => $body,
'cookie_file' => $cookieFile,
'follow_redirects' => $followRedirect,
'max_redirects' => $maxRedirects,
'ua' => $ua,
'proxy' => $proxy,
]);
if ($response['ok']) {
break;
}
$lastError = $response['error'];
// Retry only on transient errors/timeouts/5xx
if (!$this->isRetriable($response)) {
break;
}
$this->sleepMs($this->retryDelayMs);
}
if (!$response['ok'] && $lastError) {
$this->logError($url, $lastError, $response);
}
if ($this->debug) {
$this->debugDump($url, $response, [
'method' => $method,
'headers' => $headers,
'cookie_file' => $cookieFile,
'proxy' => $proxy,
]);
}
return $response;
}
// Execute cURL call
private function execCurl(string $url, array $opt): array
{
$ch = curl_init();
$reqHeaders = $this->headersToArray($opt['headers']);
// Method & body
switch ($opt['method']) {
case 'GET':
curl_setopt($ch, CURLOPT_HTTPGET, true);
break;
case 'POST':
curl_setopt($ch, CURLOPT_POST, true);
$this->applyBody($ch, $opt);
break;
default:
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $opt['method']);
$this->applyBody($ch, $opt);
}
$respHeaders = [];
curl_setopt_array($ch, [
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_CONNECTTIMEOUT => $this->connectTimeout,
CURLOPT_TIMEOUT => $this->timeout,
CURLOPT_FOLLOWLOCATION => (bool)$opt['follow_redirects'],
CURLOPT_MAXREDIRS => (int)$opt['max_redirects'],
CURLOPT_HTTPHEADER => $reqHeaders,
CURLOPT_USERAGENT => $opt['ua'],
CURLOPT_ENCODING => '', // allow gzip/deflate/br
CURLOPT_COOKIEJAR => $opt['cookie_file'],
CURLOPT_COOKIEFILE => $opt['cookie_file'],
CURLOPT_REFERER => $this->refererForUrl($url),
CURLOPT_SSL_VERIFYPEER => true,
CURLOPT_SSL_VERIFYHOST => 2,
]);
// Capture response headers
curl_setopt($ch, CURLOPT_HEADERFUNCTION, function ($curl, $header) use (&$respHeaders) {
$len = strlen($header);
$header = trim($header);
if ($header === '') return $len;
$pos = strpos($header, ':');
if ($pos !== false) {
$name = strtolower(trim(substr($header, 0, $pos)));
$value = trim(substr($header, $pos + 1));
if (!isset($respHeaders[$name])) $respHeaders[$name] = [];
$respHeaders[$name][] = $value;
} else {
$respHeaders[] = $header; // status line
}
return $len;
});
// Proxy (supports http://, https://, socks5://)
if (!empty($opt['proxy'])) {
$proxy = $opt['proxy'];
// parse scheme
$parts = parse_url($proxy);
if ($parts && isset($parts['scheme'], $parts['host'], $parts['port'])) {
curl_setopt($ch, CURLOPT_PROXY, $parts['host']);
curl_setopt($ch, CURLOPT_PROXYPORT, $parts['port']);
switch (strtolower($parts['scheme'])) {
case 'socks5':
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
break;
case 'http':
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
break;
case 'https':
// libcurl treats HTTPS proxies as HTTP + TLS tunneling; keep HTTP here
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
break;
default:
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
}
if (isset($parts['user']) || isset($parts['pass'])) {
$auth = (isset($parts['user']) ? $parts['user'] : '') . ':' . (isset($parts['pass']) ? $parts['pass'] : '');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $auth);
}
}
}
$start = microtime(true);
$body = curl_exec($ch);
$time = microtime(true) - $start;
$errno = curl_errno($ch);
$error = $errno ? curl_error($ch) : null;
$status = (int)curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
$info = curl_getinfo($ch);
curl_close($ch);
$ok = ($errno === 0) && ($status >= 200 && $status < 400);
return [
'ok' => $ok,
'status' => $status,
'body' => $body !== false ? (string)$body : '',
'headers' => $respHeaders,
'time' => $time,
'info' => $info,
'error' => $errno ? [
'code' => $errno,
'message' => $error,
] : null,
];
}
private function applyBody($ch, array $opt): void
{
if ($opt['json'] !== null) {
$payload = is_string($opt['json']) ? $opt['json'] : json_encode($opt['json'], JSON_UNESCAPED_UNICODE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
// Ensure header
// Avoid duplicates: user may have set Content-Type already
// We add only if not present
} elseif ($opt['form'] !== null) {
$payload = is_array($opt['form']) ? http_build_query($opt['form']) : (string)$opt['form'];
curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
} elseif ($opt['body'] !== null) {
curl_setopt($ch, CURLOPT_POSTFIELDS, (string)$opt['body']);
}
}
private function defaultHeaders(string $url, string $ua): array
{
$host = parse_url($url, PHP_URL_HOST) ?: '';
return [
// Browser-like core
'Host' => $host,
'Connection' => 'keep-alive',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Language' => 'en-US,en;q=0.9',
'Accept-Encoding' => 'gzip, deflate, br',
'Upgrade-Insecure-Requests' => '1',
// Some sites check sec-ch-ua headers (we mimic a Chrome-like client)
'sec-ch-ua' => '"Chromium";v="124", "Not.A/Brand";v="24", "Google Chrome";v="124"',
'sec-ch-ua-mobile'=> '?0',
'sec-ch-ua-platform' => '"Windows"',
// Content-Type is set dynamically when json/form/body present
];
}
private function refererForUrl(string $url): string
{
$parts = parse_url($url);
if (!$parts || !isset($parts['scheme'], $parts['host'])) return '';
$ref = $parts['scheme'] . '://' . $parts['host'] . '/';
return $ref;
}
private function headersToArray(array $headers): array
{
$out = [];
foreach ($headers as $k => $v) {
if (is_int($k)) {
$out[] = (string)$v;
} else {
$out[] = $k . ': ' . $v;
}
}
return $out;
}
private function mergeHeaders(array $base, array $over): array
{
// Case-insensitive merge: user headers override defaults
foreach ($over as $k => $v) {
$found = false;
foreach ($base as $bk => $_) {
if (strcasecmp($bk, $k) === 0) {
$base[$bk] = $v;
$found = true;
break;
}
}
if (!$found) $base[$k] = $v;
}
// Auto content-type if body type chosen
if (!isset($base['Content-Type'])) {
// We will set per request in applyBody; leave absent here
}
return $base;
}
private function cookiePathForUrl(string $url): string
{
$host = parse_url($url, PHP_URL_HOST) ?: 'default';
// One cookie jar per host
$path = rtrim($this->cookieDir, '/\\') . DIRECTORY_SEPARATOR . preg_replace('/[^A-Za-z0-9._-]/', '_', $host) . '.cookiejar';
if (!file_exists($path)) {
// create empty file
@touch($path);
}
return $path;
}
private function pickUserAgent(): string
{
return $this->userAgents[array_rand($this->userAgents)];
}
private function pickProxyFromFile(string $file): ?string
{
$lines = @file($file, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
if (!$lines || count($lines) === 0) return null;
// Sanitize and random pick
$candidates = [];
foreach ($lines as $line) {
$line = trim($line);
if ($line === '') continue;
// Accept formats:
// - host:port (assume http)
// - scheme://host:port
// - scheme://user:pass@host:port
if (strpos($line, '://') === false) {
// assume http proxy
$line = 'http://' . $line;
}
$candidates[] = $line;
}
if (empty($candidates)) return null;
return $candidates[array_rand($candidates)];
}
private function isRetriable(array $resp): bool
{
if ($resp['error']) {
// network errors / timeouts
$code = (int)$resp['error']['code'];
$transientErrnos = [
CURLE_COULDNT_RESOLVE_HOST,
CURLE_COULDNT_CONNECT,
CURLE_OPERATION_TIMEDOUT,
CURLE_SSL_CONNECT_ERROR,
CURLE_GOT_NOTHING,
CURLE_RECV_ERROR,
CURLE_SEND_ERROR,
];
if (in_array($code, $transientErrnos, true)) return true;
}
// Retry on 5xx responses
$status = (int)$resp['status'];
if ($status >= 500 && $status < 600) return true;
return false;
}
private function sleepMs(int $ms): void
{
usleep($ms * 1000);
}
private function logError(string $url, array $error, array $resp): void
{
$line = sprintf(
"[%s] URL=%s status=%s errno=%s msg=%s time=%.3f UA=\"%s\"\n",
date('Y-m-d H:i:s'),
$url,
(string)$resp['status'],
(string)$error['code'],
(string)$error['message'],
(float)$resp['time'],
isset($resp['info']['request_header']) ? '' : '' // keep minimal
);
@file_put_contents($this->logFile, $line, FILE_APPEND);
}
private function debugDump(string $url, array $resp, array $ctx): void
{
$snippet = substr($resp['body'], 0, 500);
$reqHeaders = $ctx['headers'];
echo "=== DEBUG REQUEST ===\n";
echo "URL: {$url}\n";
echo "Method: {$ctx['method']}\n";
echo "Proxy: " . ($ctx['proxy'] ?: 'none') . "\n";
echo "Cookie jar: {$ctx['cookie_file']}\n";
echo "Headers:\n";
foreach ($reqHeaders as $k => $v) {
echo " {$k}: {$v}\n";
}
echo "=== DEBUG RESPONSE ===\n";
echo "OK: " . ($resp['ok'] ? 'true' : 'false') . "\n";
echo "Status: {$resp['status']}\n";
echo "Time: " . number_format($resp['time'], 3) . "s\n";
echo "Resp headers:\n";
if (is_array($resp['headers'])) {
foreach ($resp['headers'] as $k => $v) {
if (is_array($v)) {
echo " {$k}: " . implode(', ', $v) . "\n";
} else {
echo " {$v}\n";
}
}
}
echo "Body (first 500 bytes):\n";
echo $snippet . "\n";
echo "======================\n";
}
}
Notes
- Cookie jars are stored per host in cookie_dir, auto-created if missing.
- Proxy list supports:
- host:port (assumed http)
- scheme://host:port
- scheme://user:pass@host:port
- Example entries:
- 1.1.1.1:10023
- socks5://1.1.1.1:10023
- http://user:[email protected]:8080
- To send JSON, pass json => array or json => string; to send forms, pass form => array.
- For raw bodies (e.g., protobuf), use body => string and set appropriate Content-Type in headers.
- If a site requires exact Chrome headers, you can override defaultHeaders via headers in request options.
If you want, I can add automatic Content-Type handling and a convenience method like get(), postJson(), postForm() that wrap request() with cleaner signatures.