Recently at Rakuten we successfully renewed our mainframe from COBOL to Java (an old Fujitsu to Oracle Exalogic) - press release. It was almost a 3 year long project and towards the end, the entire company was working on it. I transitioned into the development team for the mainframe, to rewrite several batches from Cobol to Java.

Now of course I cannot read COBOL (although its not that different). We had a machine translation software convert the cobol to java but honestly, that was hardly anything. The converted code was not smart, it used zero OOP concepts, it used customized stacks to remember function calls (haha, imagine writing your own implementation to remember function calls) and it looked like COBOL just changed its clothes. As a result, these batches performed very slow - their performance times significantly higher than their COBOL counterparts for the same data.

So the job was simple - to make these batches faster, in fact, much faster than their COBOL counterparts, otherwise what's the point?

My team chose Apache Spark as a framework to work parallelly with data. In this post, I am trying to explore why they made this decision and how Spark looks like compared to traditional batch processing.

Background of our batches:

For most of the batch processing we did, there was a fixed pattern to be noticed:

  1. Read data from database and/or file
  2. Process that data for each row of the database table/ each record of the file
  3. Write to database and/or file.

Assume we have an input file which needs to be processed and written into an output file. Assume we have 100 records in the input file.
Traditional Processing
If we were to use traditional processing, we would read the first record, process it, write it to output. Then we would read next record, process it, write to output. We would keep repeating this until all 100 rows. This is a single flow - and as we can guess, it would take us a long time to complete the whole operation. Here's how it can be represented:

Batch Processing

However we already use batch processing. We divide our input records into groups. Let's say groups of 20. That makes a total of 5 groups.
Here we read the first group together (i.e. 20 records), process them, write them to output. Then we would read the next group, process them, write them. We would keep repeating this for all 5 groups. Of course it is much faster to do this than traditional processing, since we save a lot of time switching between tasks and dealing with files. Something like this:

Parallel Processing

What if, we use that same batch processing model as above but with a twist. Let's say you had 5 different machines. You could give each group to one machine. The total time taken would be one fifth that of the batch processing model. This approach is called parallel processing. Roughly, instead of one worker, you have multiple workers who are working in batches:
This is also the approach that Spark takes to process our batches. The working of Spark is slightly different than the diagram above but we will come to the details later, in the next post.

So, why Spark?

In conclusion, we can say that indeed batch processing is much faster than traditional processing. Further, the reason why parallel processing is faster than batch processing is because you operate with multiple groups at the same time instead of just one group at a time. Of course this means we need more resources to achieve this (e.g.: the number of workers required increases in the above diagrams).
Batches which were being run on our mainframe, had very little logic to them. It was about feeding data from one process to another based on a bunch of conditions. Of course what I say is a very very simplified version of how it really looks like. But in summary, there was more work to do in getting the input and writing the output than there was in processing the data in between. Spark fit this choice because, even if we just got rid of the bloated processing of the converted code, we would have practically made no progress with the I/O operation. The logic processing would have pretty much remained the same. Apache Spark blessed us with the power of forgetting about the logic, and concentrating only on the I/O to speed up the processing times. It came with its own challenges, like writing the custom file readers and writers but that's the story of another post!

Nights like these which never cease
When the warm summer air refuses to hug
Tickles of sweat shining on forearms
The noise of the silence across the neighborhood
An array of mess waiting to be cleared
Bullet things to do piling on top of other
No drop of saliva wetting a mum mouth
Dishes lying unwashed, laundry won't do itself
A half eaten dinner won't fill up an empty heart

Misses the mere presence of you
Hibernating under the heater
Dragging through the dusky days
Wriggling out of winter's wrath
Pulling back from depressing emotions
Crawling out of the sleepy snow
And sobbing into the sleepless froids
Stepping into sunny springs
Awaiting the pink blossoms
Drinking beer to live a laugh
Finding reasons to go to the beach
Shedding off the excess fur
Dressing up in hats and gowns
Cheers to longer days and warmer spirits!

As young kids to learn about roaches
how they've survived
thousands of years
Much before the humans evaded
those days when the dinosaurs thrived
they lived,
in their tiny little bodies with hard rock armors
The ultimate survivors of long lasting famines
when food was scarce, they ate their own
cannibals such
Those pesticides and medicines
they swallowed them all
nothing can destroy them
remnants of repellents they are
In need of no homes,
no particular demands
ugly looking creatures, as strong they are
In awe we remained
of the yet scary resistant mortals
mysterious experienced and immune.

As adults to realize
the cockroaches aren't scary no more
they've emerged far more victorious
than science textbooks care to mention
In men lie their horcrux
in their ability to advance
despite worlds falling apart
Shields of ignorance strengthened by recurrence
attention spans too short to notice embellishments
desires so selfish and hearts merciless
the added audacity to take advantage of the weaker rest
Abusing the abstract, defining their own
men of powerful stature lurking in homes
Not found in corners of pipes but
relaxing in their immune chairs.
As a part of Rakuten's IT School Program, I went across 4 schools in different parts of Japan to teach high school students of about age 15-16 to teach them about Cross Border Trading, in English!

How the schools looked like

In general the school structures are the same across the country,
There are huge playgrounds in the compound and dedicated play areas for different games - football fields, tennis and badminton courts being pretty popular. Once you enter the main building, it is totally silent. It is hard to imagine you're standing in a school.

There's the main office where we would generally collect a visitor's badge. Just beside, there's a shoe-rack. You put your own shoes in there and wear dedicated slippers inside the building, strictly.
The walls of the schools are covered in posters, advertisements, artworks, manga characters throughout the building. Once you enter the classroom, you change your slippers again. The students wear only socks inside, while teachers and guests keep their pair of previous slippers inside a rack and wear another fresh pair inside. Beside each rack there are several utilities to clean the rooms. Mops, floor washes, brooms etc.

Inside Classrooms

The classrooms are fully equipped! For a stationery craver like me, it was heaven! All kinds of colors, pens, sketch pens and the list is endless. They have a good supply of computers, digital pens, drawing boards, tablets for each student! Its a fantastic environment to be in!

Since all schools are Japanese medium, all routines were conducted in Japanese. Before the class begins, the monitor instructs everyone to stand up, shout out a loud greeting yoroshiku onegaishimasu (literally means "Please help us") and bow down.

During class

This was especially challenging and difficult for me, since my lecture was in English but all schools were conducted in Japanese. I was afraid that communication would be a problem and they wouldn't understand what I speak. But thankfully, I managed by using simple words, reiterating important/difficult ideas and using Japanese words in between to keep it bound together.
Classrooms are very liberal in a way, the teacher doesn't shout or wake up a kid if he is sleeping. But the students in general were very fascinated, for some of them were being taught by a foreigner the first time and for others, they were meeting an Indian the first time!

My class in Okinawa was particularly comfortable since the students were far more accustomed to English than the others. They even asked questions without hesitation and some of them even tweeted about me at the end of the lecture!

I also took two classes in 2 schools of Aichi prefecture, which is about 2-3 hours from Tokyo, where students were more shy. They however didn't hesitate to ask me about Indian culture and latest trends in India! I somehow could feel the difference of students from Okinawa and from Aichi in terms of how shy they were, how mischievous or friendly they got and mostly the kind of questions they chose to ask me!
A few girls told me about the trending app to take pictures from and then tweeted this picture with me:

My final lecture was in Nagoya. You could tell the kids had a mix of rural humility combined with modern thinking. They seemed far more ambitious and skilled than the other kids. Although they lacked English skills, they were tremendously good at presenting and making funny sales pitches.

Some of the feedback I received at the end of the classes:

About students

I would say, the teachers like to imagine that their kids are shy, but given the comfort and ease, they very quickly jump to asking questions and try to talk in English. One more thing to notice was how much they cared about latest trends and fashion. I guess it is cool to be a high school student listening to popular music and wearing fancy clothes or using the coolest apps.

In general, the kids are highly skilled at drawing and sketching. Brilliant anime characters, ease of use of Photoshop and the comfort with which they used their pens and colours was very exciting for me. Even in their pranks with other kids, they'd make funny faces of their pictures and set them as desktop wallpapers. In all, they are full of creative design skills and posses a good sense of expressing their thoughts across.

How I feel about it!

It was a tremendous experience! To have walked into schools, met students, given them a foreigner's perspective on web design, teach them a thing or two about Japan's economy and share my piece of thoughts! Most important take away for me, was to be able to communicate clearly in English with non-native speakers! I have thoroughly enjoyed the special attention I received at all schools! Cheers!

They are talking about the randomest shit possible the moment you spot them. They are already engrossed into their deep conversation about the fragrance in the air and how its Chinese version available in the streets of Borivali for so cheap. 
I'm speaking of - a small section of the Indian community - Gujaratis.

I've taken numerous flights and encountered plenty of them, and never ever do they fail to meet my expectations of a sleepless flight. Because when God blesses us with a mouth and high shrilled voice, we ought to speak our lungs out, right? 

Believe me, Gujaratis can go on chit chatting non stop for hours. About nothing significant. Their pitched voice runs like a chant in your head. You don't want to listen it, but you are. In fact, now you know everything super personal about them. And if you happen to have your seat besides a Gujarati family, you are doomed. By the end of it, you'll know what the mother cooked for breakfast, how many theplas the kid ate, how many she packed for the flight owing to her insecurities about the flight food, how she was perfectly right about it and numerous other things you could have lived well without knowing at all. 

But I'll give one thing to them - their fathers are perhaps the best fathers in the world.  Allow me to give you some background - 
as a kid, I was so curious. About so many things. I was constantly nagging my mom and dad with questions. If it was a public place, mum either whispered the answer in my ear or she simply asked me to wAit for her explanation until we got home. But my dad, he always ignored my silly questions. He'd mostly sit with me on Sundays to talk about science stuff and teach me a thing or two. No surprises, my past questions always popped up amongst them. But at the moment, he'd ignore me. He'd pretend to have unheard anything and just continue reading the newspaper or some tax-related book. 

But, but, our Gujarati fathers. They will always entertain any question asked by their kid. Any. They are ready with 50 lines of explanation, which starts at the very basic presumptions a kid 4 years younger than their kid would have. They'll take 10 lines to reach the question, 20 to answer the question, 10 to conclude and by the next 10 lines they are talking about something else altogether. Wow. Tremendous patience and will to educate their child with their own knowledge. 

Reminds of how jobless the men in Hindi TV serials are..

You just cannot sleep even if the lights in the airplane have dimmed down. Because you're busy listening to the explanation of what the lady serving food in the air craft is called followed by how the dad once saw a horse in the middle of the street in Ahmedabad. 

The Gujarati family is determined to not let you or anyone near you sleep.

And you know what gets worse? Put a bunch of Gujaratis with another bunch of Chinese in one bus. 

This happened to me on my flight from Shanghai to Delhi. We had to take a shuttle to board the flight. And unlike those short trips, this was reaaaally long. The entire shuttle, as you can imagine was full of Indians and Chinese. Since I cannot comprehend the language, all the Chinese jibber jabber went on like a "chi chi chi chu cha chau" in my head. Constant noise in the back of my mind. Like a background score for all my thoughts. But, I had no thoughts. What remained of my brain's concentration memory block was completely occupied by 6 Gujarati men discussing about the politics in Gujarat. Since I understand Gujarati, their conversation was my mainstream thought. Shit. 20 mins in pure mess.

The nonstop banter of these six gentlemen gave me a serious headache as unfortunately my seat was next to them in the adjacent aisle. They bought alcohol and Gujarati food with them and talked as they ate. After the entire plane finished dinner and the lights were turned off, they opened 3 dakes of cards. To play. They did not play in silence. I closed my eyes, tried hard to sleep. But I just couldn't. At every step I knew which uncle was betting how much  and what the outcome was at the end of the hand. 

While I was wondering, is it only me? My young tomodachi (Japanese for friend) next to me asked why Indians were so talkative. I just shrugged. We both tried to sleep for the next 45 mins until I remember we were no longer checking up on each other. 

We even tried, to politely ask them to play a little more quietly. It worked, for a minute or two. And just when we thought we succeeded, Bhavesh uncle had already declared his winning hand, ready for some loud fun. 

I realise as I say this, no doubt the Gujaratis are the ones who have the maximum fun in their definition. They care about nothing, no one and will do what they are doing with super confidence as if it's the right thing to do. But it doesn't hurt to shut up for a while or whisper for a little talk in order to be considerate to fellow passengers. 

I made a horrendous mistake of forgetting my earphones In the recent trip, but now I'm  all equipped And ready for my flight from Delhi to Shanghai. I will sleep.