About
Activity
2K followers
Experience & Education
Projects
-
Databus (Highly Available, Highly Scalable data pipeline for INMOBI)
-
Worked on various components of the Databus project, including scribe (the backbone of Databus data flow), thrift (the backbone of scribe), Operability for the whole project. The work includes:
1. Instrumenting/Augmenting scribe to expose more useful metrics.
2. Augmenting scribe to behave correctly w.r.t operability e.g. signal handling, *really* graceful shutdown etc.
3. Patching the thrift Framed Transport to fix the duplicate messages problem. ( https://github.com/praddy/thrift…Worked on various components of the Databus project, including scribe (the backbone of Databus data flow), thrift (the backbone of scribe), Operability for the whole project. The work includes:
1. Instrumenting/Augmenting scribe to expose more useful metrics.
2. Augmenting scribe to behave correctly w.r.t operability e.g. signal handling, *really* graceful shutdown etc.
3. Patching the thrift Framed Transport to fix the duplicate messages problem. ( https://github.com/praddy/thrift )
4. Augmenting the producer side client to take "acks" into account.
5. Working with the Ops team and suggesting optimal ways to monitor the project in production.Other creatorsSee project -
Fraud Detection Service
-
Kickstarted the concept of a separated dedicated offline Fraud Detection Service to remove the burden off of the online front-end servers. A very capable engineering team took it further and got it through to production. This helped in reducing the response times of the front-end servers and by removing the online storage required we were able scale the front-end servers out across regions.
Other creators -
imetalib (domain specific ORM with an adapter for Redis)
-
As the name suggests, it is an ORM (domain specific, written in pure Ruby) for metadata services (about production servers) consisting of following entities:
1. Node Facts (cpu, memory, free memory etc.)
2. Node Metadata (cluster, partition, product etc.)
3. Node Volatile Facts (facts with TTL e.g. for use in auto deployment, alerting etc.)
Currently a UI layer and a command-line is being written around it which will facilitate inventory management and a metadata store for…As the name suggests, it is an ORM (domain specific, written in pure Ruby) for metadata services (about production servers) consisting of following entities:
1. Node Facts (cpu, memory, free memory etc.)
2. Node Metadata (cluster, partition, product etc.)
3. Node Volatile Facts (facts with TTL e.g. for use in auto deployment, alerting etc.)
Currently a UI layer and a command-line is being written around it which will facilitate inventory management and a metadata store for use with monitoring, auto deployment tools/frameworks. -
iMon (generic, scalable, pluggable, decoupled monitoring framework)
-
Designed and Architected iMon. And also helped in implementing the design. Written in Perl + C, iMon maps the monitoring process back to basics of computing:
Producing Data -> Collecting Data -> Processing Data -> Rendering Processed Data (or Producing Data to be consumed by next round)
And all this in a very generic and pluggable way. On top of iMon we wrote Metrics Aggregator (as a Processor) which supports average, moving average, histograms, 90th percentile etc. as…Designed and Architected iMon. And also helped in implementing the design. Written in Perl + C, iMon maps the monitoring process back to basics of computing:
Producing Data -> Collecting Data -> Processing Data -> Rendering Processed Data (or Producing Data to be consumed by next round)
And all this in a very generic and pluggable way. On top of iMon we wrote Metrics Aggregator (as a Processor) which supports average, moving average, histograms, 90th percentile etc. as aggregation functions, more can be added easily. We also wrote Tagger (as a Processor) which includes a power rule engine which can be used to tag the messages and then act upon tags e.g. alert, route etc.Other creatorsSee project -
Augmented Puppet for Metadata features
-
Added a new Puppet Domain Specific Language construct to populate metadata about nodes while leveraging the full feature set of the Puppet Domain Specific Language. Now puppet had a way to tag or add metadata (like cluster info, rotation info, alerting info etc.) to nodes which then can be used by various tools/frameworks to achieve goals like automated deployment, seamless migration, smart alerting/monitoring etc. Its currently in use in production.
-
Apex DOTS Pipeline implementation and Migration of Services.
-
Lead the effort of implementing a new DOTS Pipeline across several regions/datacentres and migration of services from older pipeline instance(s). DOTS (DistrO Transformation Server) Pipeline is the Highly Available & Highly Scalable backbone for all the advertisers' and publishers' updates from the UI layer.
Other creators -
Contributed and enhanced various in-house frameworks/tools
-
These included:
1. Monitoring Framework (pure Perl)
2. Generic Multiplexed TCP/SSL library (Perl)
3. Coding standard for operations team.
-
Virtual Clusterization of Bittorrent Peer Network
-
The problem was "How to keep the benefits of Bittorrent while removing the network noise created by it?" because Bittorrent started to choke our switches and Load Balancers. Designed and implemented a peerlist filter mechanism inside the Bittorrent tracker while keeping following things in mind:
1. No changes to tracker protocol to ensure any client can be used thereafter.
2. No changes to the peer-wire protocol to ensure cross-client operability.
3. Changes to minimal components…The problem was "How to keep the benefits of Bittorrent while removing the network noise created by it?" because Bittorrent started to choke our switches and Load Balancers. Designed and implemented a peerlist filter mechanism inside the Bittorrent tracker while keeping following things in mind:
1. No changes to tracker protocol to ensure any client can be used thereafter.
2. No changes to the peer-wire protocol to ensure cross-client operability.
3. Changes to minimal components in the system.
On top of the framework, implemented a subnet peerlist filter to virtually partition the large Bittorrent cluster in subnet based networks. Switches and Load Balancers doing around 70-80% cpu came down to 5-10% -
Operability for Dispaly Advertising Products
-
Reviewed all the backend products for Display Advertising Group and suggested:
1. Instrumentation Points
2. Way to Monitor
3. Architecture Enhancements
4. Ways to remove SPOFs
to make sure all the products are highly operable. -
Lead the Indian Chapter of APG Operations' Platform Group
-
Lead the Indian chapter of the APG (Advertiser Publisher Group) Operations' Platform Group. Included reviewing, designing, implementing and project managing generic ideas,frameworks and tools.
-
ERAF (Email Reporting and Assertion Framework)
-
As the sole member of the QA Automation team in India, I designed and implemented the ERAF. ERAF was essentially a Domain Specific Language with a platform for receiving and storing emails + reporting features written in pure Python. It was intended to QA the gateway level Email Security Appliance from Secure Computing (later acquired by McAfee). It helped the QA team to define workflows of test cases, which made the QA process very effective and time efficient by a factor of 10.
Other creators -
Swiss Army Knife for Emails
-
Wrote a very powerful email tool for the command-line in pure Python. Again for the QA team this tool cloud:
1. Dissect stored (in any storage format) or received emails.
2. Forge thousands of different emails per second and then either send them to single/multiple targets or store them in the desired storage format for later use.
3. OR a combination of all the above just from a single simple command-line.
This helped the QA team to conduct lengthy complex tests in minutes…Wrote a very powerful email tool for the command-line in pure Python. Again for the QA team this tool cloud:
1. Dissect stored (in any storage format) or received emails.
2. Forge thousands of different emails per second and then either send them to single/multiple targets or store them in the desired storage format for later use.
3. OR a combination of all the above just from a single simple command-line.
This helped the QA team to conduct lengthy complex tests in minutes rather than days. QA team went time efficient by a factor of 10. -
Cybernetra Mail Server
-
See projectAgain putting control in the Enterprises' hands, the ones who care about it. Architected Cybernetra MailServer to be based on various successful open source technologies like postfix, spamassassin, LDAP + a very powerful properietary rule engine which plugged into various stages of email flow throughout the system. The rule engine introduced the "Policy Based Mailing" concept which put the Enterprises' right in control.
-
Cybernetra Internet Management System
-
Gateway level Internet Management System for Enterprises. Rearchitected the whole stack to perform better in terms of throughput as the number of per user management rules on server side grow larger. This included, C development, Analyzing the Linux Kernel network stack, Analyzing layer 3 and above protocols, Analyzing various classfull and classless bandwidth management algorithms, optimizing performance of Squid and intergrating Squid with MySQL. This role included managing a team of varying…
Gateway level Internet Management System for Enterprises. Rearchitected the whole stack to perform better in terms of throughput as the number of per user management rules on server side grow larger. This included, C development, Analyzing the Linux Kernel network stack, Analyzing layer 3 and above protocols, Analyzing various classfull and classless bandwidth management algorithms, optimizing performance of Squid and intergrating Squid with MySQL. This role included managing a team of varying skillsets.
Other creatorsSee project
Recommendations received
9 people have recommended Pradeep
Join now to viewOther similar profiles
Explore collaborative articles
We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
Explore MoreOthers named Pradeep Jindal
62 others named Pradeep Jindal are on LinkedIn
See others named Pradeep Jindal