Home » Java » Problem scraping website using Java Jsoup, website not "scrolling"-Exceptionshub

Problem scraping website using Java Jsoup, website not "scrolling"-Exceptionshub

Posted by: admin February 25, 2020 Leave a comment

Questions:

My question is about the possibility of scraping data from particular websites.
At the moment, my algorithm is converting HTML to text, then checking for flagged words that are contained in a file,
and summing up the number of flags.

My problem lies in the inability to “scroll” down while scraping the website.
As you can see, it’s checking the number of flags on a twitter account, but it is limited to 50sh latest tweets.
I hope I made myself clear.

ps: I gave twitter as an example, and I’m not looking for something specific for twitter, but something more robust.

I will greatly appreciate any hints.

Sample output:

DHS & Other Agencies: 0 instances

Domestic security: 1 instances

HAZMAT & Nuclear: 0 instances

Health Concern + H1N1: 0 instances

Infrastructure Security: 2 instances

Southwest Border Violence: 1 instances

Terrorism: 0 instances

Weather/Disaster/Emergency: 2 instances

Cyber Security: 0 instances

TOTAL FLAGS: 6 instances

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

import java.io.*;
import java.util.LinkedList;
import java.util.List;

public class Main {

static List<String> dhsAndOtherAgencies = new LinkedList<>();
static List<String> domesticSecurity = new LinkedList<>();
static List<String> hazmatNuclear = new LinkedList<>();
static List<String> healthConcern = new LinkedList<>();
static List<String> infrastructureSecurity = new LinkedList<>();
static List<String> southwestBorderViolence = new LinkedList<>();
static List<String> terrorism = new LinkedList<>();
static List<String> weatherDisasterEmergency = new LinkedList<>();
static List<String> cyberSecutiry = new LinkedList<>();
static String stream;



public static void main(String[] args) throws IOException {



    createLists();
    createStream();
    Raport raport = generateReport();
    System.out.println(raport);

}

static int flagStream(List<String> list, String stream){

    int counter = 0;
    for (String flag: list){
        if(stream.contains(flag)) {
            System.out.println(flag);
            counter++;
        }
    }

    return counter;
}

static Raport generateReport(){
    return new Raport(
            flagStream(dhsAndOtherAgencies,stream),
            flagStream(domesticSecurity,stream),
            flagStream(hazmatNuclear,stream),
            flagStream(healthConcern,stream),
            flagStream(infrastructureSecurity,stream),
            flagStream(southwestBorderViolence,stream),
            flagStream(terrorism,stream),
            flagStream(weatherDisasterEmergency,stream),
            flagStream(cyberSecutiry,stream)

    );

}
static void createStream() throws IOException {
    Document doc = Jsoup.connect("https://twitter.com/realDonaldTrump").userAgent("mozilla/17.0").get();
    stream = doc.text();
}
static void createLists() throws IOException {
    BufferedReader read = new BufferedReader(new FileReader("clearListAllCases.txt"));

    String input;

    int hashCounter = 0;

    while((input=read.readLine())!=null){
        if(input.charAt(0)=='#'){
            hashCounter++;
            continue;
        }
        switch (hashCounter){

            case 1:
                dhsAndOtherAgencies.add(input);
                break;
            case 2:
                domesticSecurity.add(input);
                break;
            case 3:
                hazmatNuclear.add(input);
                break;
            case 4:
                healthConcern.add(input);
                break;
            case 5:
                infrastructureSecurity.add(input);
                break;
            case 6:
                southwestBorderViolence.add(input);
                break;
            case 7:
                terrorism.add(input);
                break;
            case 8:
                weatherDisasterEmergency.add(input);
                break;
            case 9:
                cyberSecutiry.add(input);
                break;
        }
    }
}
}

 class Raport {

int a,b,c,d,e,f,g,h,i;
int totalFlags;

Raport(int a, int b, int c, int d, int e, int f, int g, int h, int i){
    this.a = a;
    this.b = b;
    this.c = c;
    this.d = d;
    this.e = e;
    this.f = f;
    this.g = g;
    this.h = h;
    this.i = i;
    totalFlags = a+b+c+d+e+f+g+h+i;
}



public String toString(){

    return "DHS & Other Agencies:\t\t\t"+a+" instances\n"+
            "Domestic security:\t\t\t\t"+b+" instances\n"+
            "HAZMAT & Nuclear:\t\t\t\t"+c+" instances\n"+
            "Health Concern + H1N1:\t\t\t"+d+" instances\n"+
            "Infrastructure Security:\t\t"+e+" instances\n"+
            "Southwest Border Violence:\t\t"+f+" instances\n"+
            "Terrorism:\t\t\t\t\t\t"+g+" instances\n"+
            "Weather/Disaster/Emergency:\t\t"+h+" instances\n"+
            "Cyber Security:\t\t\t\t\t"+i+" instances\n"+
            "TOTAL FLAGS:\t\t\t\t\t"+totalFlags+" instances";
}

}

How to&Answers:

I would suggest opening the browser’s developer tab for trying to find out which url/endpoint the website uses for fetching new items for the infinite scroll, as JSoup does not execute Javascript itself. Then you can call the endpoint with JSoup and parse the results.

In case it does not work, It would be probably better to move to HtmlUnit or Selenium as both of them are full-featured web browser APIs which you can control with Java.