c# - 网站解析 - webbrowser 或 httpwebresponse

Question

当我试图从我的银行网站解析一些数据时遇到了一些困难。基本上，我想每天自动导出我的交易历史记录，但网上银行本身没有任何自动功能。 我目前正在试验如何模拟填写表单并点击进入下载页面并获取可用于解析的 CSV 文件。

我尝试了不同的方法，但都没有成功，请指引我到正确的路径。

 public static void getNABLogin()
    {
        try
        {
            Console.WriteLine("ENTER to begin");
            //Console.ReadLine();
            System.Net.HttpWebRequest wr = (System.Net.HttpWebRequest)System.Net.WebRequest.Create("https://ib.nab.com.au/nabib/index.jsp");
            wr.Timeout = 1000;
            wr.Method = "GET";
            wr.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36";
            wr.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
            wr.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");
            wr.Headers.Add("Accept-Encoding", "gzip,deflate,sdch");
            //wr.Connection = "Keep-Alive";
            wr.Host = "ib.nab.com.au";
            wr.KeepAlive = true;

            wr.CookieContainer = new CookieContainer();

            //////////This part will get me to the correct login page at least////////////////////
            // System.IO.Stream objStreamReceive ;
            // System.Text.Encoding objEncoding;
            // System.IO.StreamReader objStreamRead;
            // WebResponse objResponse;
            //string strOutput = string.Empty;

            //objResponse = wr.GetResponse();
            //objStreamReceive = objResponse.GetResponseStream();
            //objEncoding = System.Text.Encoding.GetEncoding("utf-8");
            //objStreamRead = new StreamReader(objStreamReceive, objEncoding); // Set function return value
            //strOutput = objStreamRead.ReadToEnd();
            ///////////////////////////////
            System.Net.HttpWebResponse wresp = (System.Net.HttpWebResponse)wr.GetResponse();

            System.Windows.Forms.WebBrowser wb = new System.Windows.Forms.WebBrowser();

            wb.DocumentStream = wresp.GetResponseStream();
            wb.ScriptErrorsSuppressed = true;

           wb.DocumentCompleted += (sndr, e) =>
            {
                /////////////After dumping the document text into a text file, I get a different page/////////////////
                //////////////I get the normal website instead of login page////////////////////////
               System.IO.StreamWriter file = new System.IO.StreamWriter("C:\\temp\\test.txt");
               Console.WriteLine(wb.DocumentText);
               file.WriteLine(wb.DocumentText);
               System.Windows.Forms.HtmlDocument d = wb.Document;

               System.Windows.Forms.HtmlElementCollection ctrlCol = d.GetElementsByTagName("script");
               foreach (System.Windows.Forms.HtmlElement tag in ctrlCol)
               {
                   tag.SetAttribute("src", string.Format("https://ib.nab.com.au{0}", tag.GetAttribute("src")));
               }


               ctrlCol = d.GetElementsByTagName("input");
               foreach (System.Windows.Forms.HtmlElement tag in ctrlCol)
               {
                   if (tag.GetAttribute("name") == "userid")
                   {
                       tag.SetAttribute("value", "123456");
                   }
                   else if (tag.GetAttribute("name") == "password")
                   {
                       tag.SetAttribute("value", "nabPassword");
                   }
                   file.WriteLine(tag.GetAttribute("name"));
               }

               file.Close();
               // object y = wb.Document.InvokeScript("validateLogin");
            };

           while (wb.ReadyState != System.Windows.Forms.WebBrowserReadyState.Complete)
           {
               System.Windows.Forms.Application.DoEvents();
           }
        }
        catch(Exception e)
        {
            System.IO.StreamWriter file = new System.IO.StreamWriter("C:\\temp\\error.txt");
            file.WriteLine(e.Message);
            Console.WriteLine(string.Format("error: {0}", e.Message));
            Console.ReadLine();
        }

我从一个线程调用了这个方法(因为您可能知道 webbrowser 需要成为 STA 线程才能工作)。 如代码中所述，我使用 httpwebresponse 方法正确获得了登录页面。但是当我尝试使用文档流加载到网络浏览器时，我进入了另一个网站。

下一个问题是，进入登录页面后我应该怎么做，如何模拟点击和填充数据(我目前的理论是尝试使用httpwebrequest发布一些数据)。

请阐明这一点。非常感谢任何意见或信息。 非常感谢您。

Answer 1

你可以像浏览器一样使用 selenium 并转到你想去的地方并使用 HtmlAgilityPack 解析页面。两者都有 c# 支持。非常简单的控制台应用程序可以休息

Selenium

http://www.seleniumhq.org/docs/02_selenium_ide.jsp#chapter02-reference

HtmlAgilityPack https://htmlagilitypack.codeplex.com/wikipage?title=Examples

您可以使用 selenium 和 c# 填写表单并像这样发布

//Navigate to the site
 driver.Navigate().GoToUrl("http://www.google.com.au");
 // Find the text input element by its name
 IWebElement query = driver.FindElement(By.Name("q"));
 // Enter something to search for
 query.SendKeys("Selenium");
 // Now submit the form
 query.Submit();
 // Google's search is rendered dynamically with JavaScript.
 // Wait for the page to load, timeout after 5 seconds
 WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(5));
 wait.Until((d) => { return d.Title.StartsWith("selenium"); });

您可以使用 HtmlAgility 像这样解析数据(此示例表)

var cols = doc.DocumentNode.SelectNodes("//table[@id='table2']//tr//td");
for (int ii = 0; ii < cols.Count; ii=ii+2)
{
    string name = cols[ii].InnerText.Trim();
    int age = int.Parse(cols[ii+1].InnerText.Split(' ')[1]);
}