There is this fish in the Great Barrier Reef of Australia - tuskfish. 

He wakes up every morning and swims to the edge of the reef. Then he digs the corals and the rubble searching for something. It looks unnatural to watch something dig with its fins, no hands. But it continues, and then finally he finds a clam. He manages to hold it in its mouth and swim all the way back, except this time with the clam in his mouth. 

He arrives at his lunch table, its kind of a hollow coral with walls. Like a shallow well. It wastes no time, and it throws the clam’s hard white shell across the walls in an attempt to smash it. Nothing happens. He repeats, he keeps throwing the clam on the same spot with high accuracy. Not giving up. 

Finally there’s a small crack, after a hundred attempts. He throws it once more, with the same intensity and there, it breaks open. A small squiggly meat comes out and the fish devours it. The camera zooms out, you see inside the bowl shaped rock, so many - so so many of such broken shells, accumulating over days. The fish does this everyday, all its life. Its a mere fish.

What do I have to complain about? There's so much hard work to do.

Courtesy: Blue Planet II

Recently at Rakuten we successfully renewed our mainframe from COBOL to Java (an old Fujitsu to Oracle Exalogic) - press release. It was almost a 3 year long project and towards the end, the entire company was working on it. I transitioned into the development team for the mainframe, to rewrite several batches from Cobol to Java.

Now of course I cannot read COBOL (although its not that different). We had a machine translation software convert the cobol to java but honestly, that was hardly anything. The converted code was not smart, it used zero OOP concepts, it used customized stacks to remember function calls (haha, imagine writing your own implementation to remember function calls) and it looked like COBOL just changed its clothes. As a result, these batches performed very slow - their performance times significantly higher than their COBOL counterparts for the same data.

So the job was simple - to make these batches faster, in fact, much faster than their COBOL counterparts, otherwise what's the point?

My team chose Apache Spark as a framework to work parallelly with data. In this post, I am trying to explore why they made this decision and how Spark looks like compared to traditional batch processing.

Background of our batches:

For most of the batch processing we did, there was a fixed pattern to be noticed:

  1. Read data from database and/or file
  2. Process that data for each row of the database table/ each record of the file
  3. Write to database and/or file.

Assume we have an input file which needs to be processed and written into an output file. Assume we have 100 records in the input file.
Traditional Processing
If we were to use traditional processing, we would read the first record, process it, write it to output. Then we would read next record, process it, write to output. We would keep repeating this until all 100 rows. This is a single flow - and as we can guess, it would take us a long time to complete the whole operation. Here's how it can be represented:

Batch Processing

However we already use batch processing. We divide our input records into groups. Let's say groups of 20. That makes a total of 5 groups.
Here we read the first group together (i.e. 20 records), process them, write them to output. Then we would read the next group, process them, write them. We would keep repeating this for all 5 groups. Of course it is much faster to do this than traditional processing, since we save a lot of time switching between tasks and dealing with files. Something like this:

Parallel Processing

What if, we use that same batch processing model as above but with a twist. Let's say you had 5 different machines. You could give each group to one machine. The total time taken would be one fifth that of the batch processing model. This approach is called parallel processing. Roughly, instead of one worker, you have multiple workers who are working in batches:
This is also the approach that Spark takes to process our batches. The working of Spark is slightly different than the diagram above but we will come to the details later, in the next post.

So, why Spark?

In conclusion, we can say that indeed batch processing is much faster than traditional processing. Further, the reason why parallel processing is faster than batch processing is because you operate with multiple groups at the same time instead of just one group at a time. Of course this means we need more resources to achieve this (e.g.: the number of workers required increases in the above diagrams).
Batches which were being run on our mainframe, had very little logic to them. It was about feeding data from one process to another based on a bunch of conditions. Of course what I say is a very very simplified version of how it really looks like. But in summary, there was more work to do in getting the input and writing the output than there was in processing the data in between. Spark fit this choice because, even if we just got rid of the bloated processing of the converted code, we would have practically made no progress with the I/O operation. The logic processing would have pretty much remained the same. Apache Spark blessed us with the power of forgetting about the logic, and concentrating only on the I/O to speed up the processing times. It came with its own challenges, like writing the custom file readers and writers but that's the story of another post!

Nights like these which never cease
When the warm summer air refuses to hug
Tickles of sweat shining on forearms
The noise of the silence across the neighborhood
An array of mess waiting to be cleared
Bullet things to do piling on top of other
No drop of saliva wetting a mum mouth
Dishes lying unwashed, laundry won't do itself
A half eaten dinner won't fill up an empty heart

Misses the mere presence of you
Hibernating under the heater
Dragging through the dusky days
Wriggling out of winter's wrath
Pulling back from depressing emotions
Crawling out of the sleepy snow
And sobbing into the sleepless froids
Stepping into sunny springs
Awaiting the pink blossoms
Drinking beer to live a laugh
Finding reasons to go to the beach
Shedding off the excess fur
Dressing up in hats and gowns
Cheers to longer days and warmer spirits!

As young kids to learn about roaches
how they've survived
thousands of years
Much before the humans evaded
those days when the dinosaurs thrived
they lived,
in their tiny little bodies with hard rock armors
The ultimate survivors of long lasting famines
when food was scarce, they ate their own
cannibals such
Those pesticides and medicines
they swallowed them all
nothing can destroy them
remnants of repellents they are
In need of no homes,
no particular demands
ugly looking creatures, as strong they are
In awe we remained
of the yet scary resistant mortals
mysterious experienced and immune.

As adults to realize
the cockroaches aren't scary no more
they've emerged far more victorious
than science textbooks care to mention
In men lie their horcrux
in their ability to advance
despite worlds falling apart
Shields of ignorance strengthened by recurrence
attention spans too short to notice embellishments
desires so selfish and hearts merciless
the added audacity to take advantage of the weaker rest
Abusing the abstract, defining their own
men of powerful stature lurking in homes
Not found in corners of pipes but
relaxing in their immune chairs.
As a part of Rakuten's IT School Program, I went across 4 schools in different parts of Japan to teach high school students of about age 15-16 to teach them about Cross Border Trading, in English!

How the schools looked like

In general the school structures are the same across the country,
There are huge playgrounds in the compound and dedicated play areas for different games - football fields, tennis and badminton courts being pretty popular. Once you enter the main building, it is totally silent. It is hard to imagine you're standing in a school.

There's the main office where we would generally collect a visitor's badge. Just beside, there's a shoe-rack. You put your own shoes in there and wear dedicated slippers inside the building, strictly.
The walls of the schools are covered in posters, advertisements, artworks, manga characters throughout the building. Once you enter the classroom, you change your slippers again. The students wear only socks inside, while teachers and guests keep their pair of previous slippers inside a rack and wear another fresh pair inside. Beside each rack there are several utilities to clean the rooms. Mops, floor washes, brooms etc.

Inside Classrooms

The classrooms are fully equipped! For a stationery craver like me, it was heaven! All kinds of colors, pens, sketch pens and the list is endless. They have a good supply of computers, digital pens, drawing boards, tablets for each student! Its a fantastic environment to be in!

Since all schools are Japanese medium, all routines were conducted in Japanese. Before the class begins, the monitor instructs everyone to stand up, shout out a loud greeting yoroshiku onegaishimasu (literally means "Please help us") and bow down.

During class

This was especially challenging and difficult for me, since my lecture was in English but all schools were conducted in Japanese. I was afraid that communication would be a problem and they wouldn't understand what I speak. But thankfully, I managed by using simple words, reiterating important/difficult ideas and using Japanese words in between to keep it bound together.
Classrooms are very liberal in a way, the teacher doesn't shout or wake up a kid if he is sleeping. But the students in general were very fascinated, for some of them were being taught by a foreigner the first time and for others, they were meeting an Indian the first time!

My class in Okinawa was particularly comfortable since the students were far more accustomed to English than the others. They even asked questions without hesitation and some of them even tweeted about me at the end of the lecture!

I also took two classes in 2 schools of Aichi prefecture, which is about 2-3 hours from Tokyo, where students were more shy. They however didn't hesitate to ask me about Indian culture and latest trends in India! I somehow could feel the difference of students from Okinawa and from Aichi in terms of how shy they were, how mischievous or friendly they got and mostly the kind of questions they chose to ask me!
A few girls told me about the trending app to take pictures from and then tweeted this picture with me:

My final lecture was in Nagoya. You could tell the kids had a mix of rural humility combined with modern thinking. They seemed far more ambitious and skilled than the other kids. Although they lacked English skills, they were tremendously good at presenting and making funny sales pitches.

Some of the feedback I received at the end of the classes:

About students

I would say, the teachers like to imagine that their kids are shy, but given the comfort and ease, they very quickly jump to asking questions and try to talk in English. One more thing to notice was how much they cared about latest trends and fashion. I guess it is cool to be a high school student listening to popular music and wearing fancy clothes or using the coolest apps.

In general, the kids are highly skilled at drawing and sketching. Brilliant anime characters, ease of use of Photoshop and the comfort with which they used their pens and colours was very exciting for me. Even in their pranks with other kids, they'd make funny faces of their pictures and set them as desktop wallpapers. In all, they are full of creative design skills and posses a good sense of expressing their thoughts across.

How I feel about it!

It was a tremendous experience! To have walked into schools, met students, given them a foreigner's perspective on web design, teach them a thing or two about Japan's economy and share my piece of thoughts! Most important take away for me, was to be able to communicate clearly in English with non-native speakers! I have thoroughly enjoyed the special attention I received at all schools! Cheers!