Home » Java » java – FusionAuth incomplete reindex with AWS Elasticsearch-Exceptionshub

java – FusionAuth incomplete reindex with AWS Elasticsearch-Exceptionshub

Posted by: admin February 25, 2020 Leave a comment

Questions:

I am migrating from a self-hosted Elasticsearch FusionAuth search to an AWS Elasticsearch Service solution.

I have a new FusionAuth app EC2 instance reading from the in-use database that is configured to use the new Elasticsearch service.

On triggering a reindex from the new app instance I see that only around 60k or 62.5k documents are being written to the new index when I am expecting roughly 6mil.

I see no errors from AWS’s Elasticsearch Service and in the app’s logs I can see: (endpoint intentionally omitted)

Feb 13, 2020 10:18:46.116 AM INFO  io.fusionauth.api.service.search.ElasticSearchClientProvider - Connecting to FusionAuth Search Engine at [https://vpc-<<omitted>>.eu-west-1.es.amazonaws.com]
13-Feb-2020 11:19:55.176 INFO [http-nio-9011-exec-3] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
 Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
        java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
                at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:430)
                at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684)
                at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:808)
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
"/usr/local/fusionauth/logs/fusionauth-app.log" [readonly] 43708L, 4308629C                                                                                                                                                                                                                                                                               42183,1       96%
        at io.fusionauth.api.service.search.client.domain.documents.IndexUser.<init>(IndexUser.java:79)
        at io.fusionauth.api.service.search.ElasticsearchSearchEngine.lambda$index$1(ElasticsearchSearchEngine.java:140)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at io.fusionauth.api.service.search.ElasticsearchSearchEngine.index(ElasticsearchSearchEngine.java:140)
        at io.fusionauth.api.service.user.ReindexRunner$ReindexWorker.run(ReindexRunner.java:101)
        at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-14" java.lang.NullPointerException
        at io.fusionauth.api.service.search.client.domain.documents.IndexUser.<init>(IndexUser.java:79)
        at io.fusionauth.api.service.search.ElasticsearchSearchEngine.lambda$index$1(ElasticsearchSearchEngine.java:140)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at io.fusionauth.api.service.search.ElasticsearchSearchEngine.index(ElasticsearchSearchEngine.java:140)
        at io.fusionauth.api.service.user.ReindexRunner$ReindexWorker.run(ReindexRunner.java:101)
        at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-13" java.lang.NullPointerException
Exception in thread "Thread-11" java.lang.NullPointerException
Exception in thread "Thread-12" java.lang.NullPointerException
Feb 18, 2020 10:23:29.064 AM INFO  io.fusionauth.api.service.user.ReindexRunner - Reindex completed in [86797] ms or [86] seconds.

Although there are some exceptions there is also an “Reindex completed” INFO log at the end.

Not knowing the ins-and-outs of Elasticsearch I’m also not sure where to start in investigating a NullPointerException.

How to&Answers:

It looks like the re-index operation is taking an exception which is likely the cause of the truncated index.

Exception in thread "Thread-14" java.lang.NullPointerException
        at io.fusionauth.api.service.search.client.domain.documents.IndexUser.<init>(IndexUser.java:79)

This code makes an assumption that you have a username or email address. This should be enforced by the FusionAuth APIs. But in this case, for this exception to occur the email and username are both NULL.

How did you get users into the db, using the Import API, User API, or something else?

In theory you should find at least one user with a NULL value for the email and username.

This query – or similar – should find the offending user, then we need to identify how this user was added to FusionAuth.

SELECT email, username from identities WHERE email IS NULL OR username IS NULL