c# - 无法使用c#获取网站内容

Question

这是我获取网站内容的代码行:

private string GetContent(string url) {
    var request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = "GET";
    var content = String.Empty;
    HttpStatusCode statusCode;
    using (var response = request.GetResponse())
        using (var stream = response.GetResponseStream())
        {
            var contentType = response.ContentType;
            Encoding encoding = null;
            if (contentType != null)
            {
                var match = Regex.Match(contentType, @"(?<=charset\=).*");
                if (match.Success)
                    encoding = Encoding.GetEncoding(match.ToString());
            }

            encoding = encoding ?? Encoding.UTF8;

            statusCode = ((HttpWebResponse)response).StatusCode;
            using (var reader = new StreamReader(stream, encoding))
                content = reader.ReadToEnd();
        }
    return content;
}

我尝试使用链接运行这行代码:http://google.com 。一切都完成了。但是当我使用链接运行时:http://batdongsan.com.vn/ 。它不起作用并显示“抱歉!出了点问题。”。我不知道为什么会发生这样的事。我如何获取第二个链接的内容？

Answer 1

看起来该站点正在检查 User-Agent header ，并且由于默认情况下未设置它，因此它返回错误消息。我添加了浏览器发送的内容，并且能够获取该链接的内容。只需添加设置 UserAgent 的行，如下所示:

// ...
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36";

var content = String.Empty;
HttpStatusCode statusCode;
// ...