- Proposed, implemented, and open-sourced a wrapper project under the MIT license that enables real-time configuration synchronization in Python long-running processes using etcd3, eliminating the need for periodic calls every x seconds.
- Co-Led the re-architecture of services code to integrate Celery, optimizing the Mailbox processing service. Leveraged Celery’s multi-processing, auto-scaling, and load distribution features to address memory management issues, enhance fault tolerance, and reduce infrastructure costs.
- Optimized the Mailbox distribution algorithm to reduce overall email processing time and leveraged underutilized Python processes to enhance hardware resource utilization.
- Led the migration of services from Python 2.7 to Python 3.8 , designing and developing internal APIs with FastAPI while seamlessly integrating new features into existing Flask APIs. Streamlined the codebase by deprecating obsolete features and performing impact analysis to ensure the smooth operation of both legacy and new functionalities.
- Developed FNFP project scripts to extract and enrich log data from AWS CloudWatch , providing valuable insights for cross-team analysis.
- Introduced Apache Kafka into the system to enhance fault tolerance and scalability, implementing near-real-time use cases in conjunction with Redis.
- Redesigned and transitioned the services from monolithic to microservices architecture to address deployment challenges and improve codebase maintenance.
- Implemented Docker-based containerization by migrating most services to Docker and deploying multiple Redis instances within Docker containers to improve system fault tolerance and address downtime issues.
- Developed ETL Data Pipelines from scratch for threat feed processing, providing IOC to APIs at low latency using Redis caching. Utilized Python, Redis, MongoDB, PostgreSQL, and Apache Airflow to ensure efficient data handling for mobile applications and a threat intelligence platform.
- Developed a Syslogs Parsing Service from scratch to extract and analyze inbound and outbound traffic, compare it with IOCs, and push matching events over the sockets for real-time notifications to the Threat Intelligence Platform.
- Implemented MongoDB Replication to support event replication through Apache Kafka and created backup data migration scripts to transfer data from Google Cloud to on-premises storage.
- Maintained and Enhanced Django REST Framework APIs, integrating new features and performing deployments and testing. Managed SSL certificate rotation and set up a local testing environment on on-premises cloud infrastructure.
Real-time detection and loading of dynamic configuration changes from an etcd database to a Python process.
Instead of periodically pooling configurations from a database at fixed intervals, we can utilize etcd database to watch for changes on specific keys (configs). Whenever new changes occur on these keys, the etcd server will push those changes to the client, and then react on them accordingly.
Under the hood, it utilizes the python-etcd3 library, which provides a callback method to listen for changes on keys.
Here are some additions made :REST API module is for mobile application and secured from distributed denial-of-service (DDoS) attack using API rate limiter.
Redis in-memory database is used for the high availability of the application and provides very low latency.
Authentication and Authorization feature for staff members e.g. if multiple laboratories upload confirmed cases of covid19.
Compare to other rate limit algorithms sliding window counter is
Rate limit strategies allowed in this algorithm
Rate limit HTTP headers allowed in this algorithm
The reason behind building this kind of application was users were/are dramatically increasing to use social networking applications. We also didn't want to build some management system, want to explore some NoSQL and real-time database which is Google Firebase.