I’ve got myself an Amazon ec2 Micro Instance (A VPN Server) to play around with.
The problem is that Amazon charge you for every disc IO you do in a Micro Instance.
The instance is running Amazon Linux a flavor of CentOS.
I’ve started a Scala application in Play 2.0(.2) framework on the server and I’m the only one who connects to the application.
I have observed that every few second something on the server commits IO transactions, to narrow it down I installed a Linux program called
Here is an output after a couple of seconds.
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> 23333 be/4 root 0.00 B/s 11.91 K/s 0.00 % 0.00 % COMMAND java -Dsbt.ivy.home=/usr/play-2.0.2/framework/../repository -Djava.runtime.name=OpenJDK ~/jars/slf4j-api.jar:/usr/play-2.0.2/repository/local/org.slf4j/jul-to-slf4j/1.6.4/jars/j
A cat from the log file
cat /home/ec2-user/socketTest/logs/application.log 2012-07-05 11:43:31,881 - [INFO] - from play in main Listening for HTTP on port 9000...
So Play doesn’t write anything to the log file.
First question have I understood the iotop correct and that Play indeed is the disc IO thief.
If so why do play use IO?
My application is a simple websocket example. In essence it echos the input to the output. The IO occurs even thou nothing is pushed thru websockets.
I’ve finally found the answer.
By observing when Play made an IO transaction I instantly executed this command:
touch -d '-10 seconds' /tmp/newerthan find / ! -fstype proc -newer /tmp/newerthan
This returned one interesting line:
While googling on this i stumbled upon a Bug ID: 5012932 from sun JVM creates subdirectory “hsperfdata_xxx”. Java does this to enable noninvasive observability
of the JRE, they claim it’s a feature and not a bug that’s why it hasn’t been resolved.
A solution presented to disable this “feature” was to make use of an undocumented option
-XX:-UsePerfData. I tried this but unfortunately Play kept making the IO transactions.
But after some more digging I found another switch
So I executed
export _JAVA_OPTIONS="-XX:+PerfDisableSharedMem" before starting Play .
And… Voilà Play stopped making the IO transactions!
If you’d like to know which files are being written to, you can use
inotifywait, which is shipped in the
inotify-tools package (at least that’s what Fedora calls it):
$ inotifywait -r -m /opt /etc /var -e ATTRIB -e CREATE -e MODIFY -e DELETE Setting up watches. Beware: since -r was given, this may take a while! Watches established. /var/tmp/ CREATE etilqs_uOXWfa8v7DkNBgd /var/tmp/ DELETE etilqs_uOXWfa8v7DkNBgd /var/tmp/ MODIFY etilqs_uOXWfa8v7DkNBgd /var/tmp/ MODIFY etilqs_uOXWfa8v7DkNBgd /var/tmp/ MODIFY etilqs_uOXWfa8v7DkNBgd ...
Obviously, replace “/opt /etc /var” above with whatever directories you suspect are interesting.
It’s almost certainly much more efficient than running
lsof in a loop and grepping its output. But you probably shouldn’t leave it running for a long time in production.
Anyway, once you know which files are being written to, you’ll be well on the way to stopping it. 🙂
You can also recursively
ls your directory and sort by
mtime or whatever)
Yet another solution (simple but useful):
# watch df
Then you can run this to go deeper:
# du -s /your/path/