我有 CSV 文件,已存储在 Google 云存储中。我正在阅读这个 CSV 文件,并为每个出现的单词构建一个树形图。有没有办法可以排序并仅显示文件中的前 10 个单词?
这是我的代码:
@SuppressWarnings("serial")
public class GoogleCloudStorageServlet extends HttpServlet {
public static final String BUCKETNAME = "bigdata";
public static final String FILENAME = "Railways.csv";
@SuppressWarnings({ "unchecked", "rawtypes" })
@Override
public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
resp.setContentType("text/plain");
resp.getWriter().println("Hello, world from java");
GcsService gcsService = GcsServiceFactory.createGcsService();
GcsFilename filename = new GcsFilename(BUCKETNAME, FILENAME);
GcsFileOptions options = new GcsFileOptions.Builder()
.mimeType("text/html")
.acl("public-read")
.addUserMetadata("myfield1", "my field value")
.build();
GcsOutputChannel writeChannel = gcsService.createOrReplace(filename, options);
PrintWriter writer = new PrintWriter(Channels.newWriter(writeChannel, "UTF8"));
GcsInputChannel readChannel = null;
BufferedReader reader = null;
String cvsSplitBy = ",";
try {
readChannel = gcsService.openReadChannel(filename, 0);
reader = new BufferedReader(Channels.newReader(readChannel, "UTF8"));
String line;
TreeMap<String, Integer> map = new TreeMap<String, Integer>();
while ((line = reader.readLine()) != null) {
String[] post = line.split(cvsSplitBy);
String[] words = post[1].split("[ \n\t\r.,;:!?(){}]");
for (int counter = 0; counter < words.length; counter++) {
String key = words[counter].toLowerCase(); // remove .toLowerCase for Case Sensitive result.
if (key.length() > 0) {
if (map.get(key) == null) {
map.put(key, 1);
}
else {
int value = map.get(key).intValue();
value++;
map.put(key, value);
}
}
}
//Display only top 10 words in the file
}
} finally {
if (reader != null) { reader.close(); }
}
}
}
最佳答案
计算 CSV 文件中前 10 个单词的方式取决于文件的大小。
小文件(可以存储在内存中)
如果文件较小,您可以使用某种针对您的情况优化的集合(例如 Bill Lin 提到的 Multiset)或自行计算。
Map<String, Integer> counts = new HashMap<String, Integer>(); for (String word : words) { Integer count = counts.get(word); if (count == null) { counts.put(word, 1); } else { counts.put(word, count + 1); } }
如果文件确实很小,这样的计算可以在单个请求的范围内处理。
中/大尺寸文件
如果是中型或大型文件,您可能会超出请求限制(60 秒),并且可能会耗尽可用内存。效率也不会很高。您需要一种不同的方法。
尝试查看MapReduce for App Engine .
MapReduce is a programming model for processing large amounts of data in a parallel and distributed fashion. It is useful for large, long-running jobs that cannot be handled within the scope of a single request.
显示结果的方式取决于您的处理模式。
同步
如果您的计算是同步的,您可以通过传递的响应对象直接从 Servlet 显示结果。
HttpServletResponse#getWriter()
异步
如果是异步计算,您需要将结果存储在某个地方(例如Datastore)并按需显示
关于java - 显示 CSV 文件中的前 10 个单词,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/24576813/