Grails batch import slow

I working in a Grails project, and find it is a good tools, hope the Grails development will continuous.

But one day, I work on the Batch import, and find that the import is not fast as I expect (about 0.5 for 1 record), and the most hopeless is that, it become slower and slower when the batch size is larger than 1500.

Then I find a blog post, which teach us to flush and clear the session, it work a little bit, and make it faster, but the job still become slow after 1500.
Batch Import Performance with Grails and MySQL

I click Ctrl-C to stop the project, when it use 3~4 sec to import one record, and I found that there are a stack trace which related to Compass Transaction (Searchable plugins) before the service stop.

Then I try to stop the Searchable indexing before I import the record to database something like
(ref : Searchable Plugin – Methods – stopMirroring)

def searchableService

def import = {
searchableService.stopMirroring() ;
// … import data
searchableService.startMirroring()
searchableService.index()
}

And at last, don’t forget to start the mirroring again, and make the index again.
But be careful that, when more than 1 person use this function, you may need to add the stopMirroring inside the import loop.

56 comments to Grails batch import slow

Leave a Reply

 

 

 

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>