Home » Linux » Play constantly writes to disc? Causing higher bill on Amazon ec2

Play constantly writes to disc? Causing higher bill on Amazon ec2

Posted by: admin November 30, 2017 Leave a comment

Questions:

I’ve got myself an Amazon ec2 Micro Instance (A VPN Server) to play around with.
The problem is that Amazon charge you for every disc IO you do in a Micro Instance.
The instance is running Amazon Linux a flavor of CentOS.

I’ve started a Scala application in Play 2.0(.2) framework on the server and I’m the only one who connects to the application.

I have observed that every few second something on the server commits IO transactions, to narrow it down I installed a Linux program called iotop.

Here is an output after a couple of seconds.

TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    
23333 be/4 root        0.00 B/s   11.91 K/s  0.00 %  0.00 %  
COMMAND java -Dsbt.ivy.home=/usr/play-2.0.2/framework/../repository -Djava.runtime.name=OpenJDK ~/jars/slf4j-api.jar:/usr/play-2.0.2/repository/local/org.slf4j/jul-to-slf4j/1.6.4/jars/j

A cat from the log file

cat /home/ec2-user/socketTest/logs/application.log
2012-07-05 11:43:31,881 - [INFO] - from play in main
Listening for HTTP on port 9000...

So Play doesn’t write anything to the log file.

First question have I understood the iotop correct and that Play indeed is the disc IO thief.
If so why do play use IO?

My application is a simple websocket example. In essence it echos the input to the output. The IO occurs even thou nothing is pushed thru websockets.

Answers:

I’ve finally found the answer.

By observing when Play made an IO transaction I instantly executed this command:

touch -d '-10 seconds' /tmp/newerthan
find / ! -fstype proc -newer /tmp/newerthan

This returned one interesting line:

/tmp/hsperfdata_root/23320

While googling on this i stumbled upon a Bug ID: 5012932 from sun JVM creates subdirectory “hsperfdata_xxx”. Java does this to enable noninvasive observability
of the JRE, they claim it’s a feature and not a bug that’s why it hasn’t been resolved.
A solution presented to disable this “feature” was to make use of an undocumented option -XX:-UsePerfData. I tried this but unfortunately Play kept making the IO transactions.

But after some more digging I found another switch -XX:+PerfDisableSharedMem.

So I executed export _JAVA_OPTIONS="-XX:+PerfDisableSharedMem" before starting Play .

And… Voilà Play stopped making the IO transactions!

Questions:
Answers:

If you’d like to know which files are being written to, you can use inotifywait, which is shipped in the inotify-tools package (at least that’s what Fedora calls it):

$ inotifywait -r -m /opt /etc /var -e ATTRIB -e CREATE -e MODIFY -e DELETE
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/var/tmp/ CREATE etilqs_uOXWfa8v7DkNBgd
/var/tmp/ DELETE etilqs_uOXWfa8v7DkNBgd
/var/tmp/ MODIFY etilqs_uOXWfa8v7DkNBgd
/var/tmp/ MODIFY etilqs_uOXWfa8v7DkNBgd
/var/tmp/ MODIFY etilqs_uOXWfa8v7DkNBgd
  ...

Obviously, replace “/opt /etc /var” above with whatever directories you suspect are interesting.

It’s almost certainly much more efficient than running lsof in a loop and grepping its output. But you probably shouldn’t leave it running for a long time in production.

Anyway, once you know which files are being written to, you’ll be well on the way to stopping it. 🙂

Questions:
Answers:

You can also recursively ls your directory and sort by atime (ormtime or whatever)

Questions:
Answers:

Yet another solution (simple but useful):

# watch df

Then you can run this to go deeper:

# du -s /your/path/