It can be fun being a developer, but sometimes you can forget you are one and begin to think human. Today as been a somewhat long day from visiting my GP and ending up in the hospital behind a long queue of patients waiting for an X-ray. I am at my desk with a couple of hours to churn through, I decided to tackle a long pending Ticket. An hour and a few minutes later, I am done with the update but there is a second part to this ticket. I need to refresh a lot of data in a CSV file stored remotely on an AWS S3 bucket.
I download the file and decide to tweak the refresh locally. So I update the code and statically type in date range parameter. I was meant to get a year’s worth of data. We looking at some heavy MegaByte worth of JSON data. We do agree that is just suicidal to think you can transfer 80MB worth of JSON data over the wire from one HTTP request. So the task would be to break the API date range between the months of the year.
I tried breaking the request down but got a 400 – Bad Request error message. So as usual, I decided to test the boundaries of the API by reducing my range from 1 year to Monthly, Weekly and then I came upon the 1 Day interval as the ideal range to work with. So I started out by manually adjusting the data and seeing what data gets pulled. With a bit of frustration and trying to figure out how I can automate the changing of the API date range. For some reason, the thought writing new code to get the result didn’t come to me on time for some reason. I spoke to my colleague next to me, and he suggested I write a function to do all the date range update. This was a brilliant idea, I could write this code, get the effect I wanted and revert code back. We can discuss code review and why date range was hard-coded in another post.
I wonder why I didn’t think of this. Sometimes as developers, we do miss the joy of innovating, writing new codes and being innovative with solutions to a simple task. I did write the code as expected, got the result I wanted but still came across other bottlenecks which had more to do with the API than my code.
Morale of the story:
Never forget you are a developer. Always seek to innovate.