Find Jobs
Hire Freelancers

Send 1000k+ HTTP requests a second

£10-20 GBP

Cerrado
Publicado hace más de 5 años

£10-20 GBP

Pagado a la entrega
We would essentially require a solution to make > 1000 HTTP requests in parallel a second. Our proxy provider allows us to use residential proxies with an incredibly low latency. Thus, speed is not a bottleneck in the given use-case nor should it be whatsoever. In terms of a programming language, we would prefer Golang or Python to be used. However, If you have anything else in mind then we're open to ideas! The plan would be to have a solution which makes > 1000 HTTP requests in parallel a second while connected to a specified HTTP proxy (Oxylabs). It is an absolute requirement that we meet these specifications and have the resources to do so. We have already looked into this topic but failed with little success due to host connection limitations or improper techniques used while attempting the given task. Thus, we have attempted to reach out to the world of Freelancer. The endpoint (i.e. [login to view URL]) in mind is not to change although the payload (data) sent shall change with each request. All required additional information shall be given upon request. If you think that you could assist us from a long-time perspective then please do let us know as there is an opportunity to make a fixed amount of $1000-2000 every week. See this as more-so a test of knowledge then anything. We have an example version of what we need done but it sadly has a bottleneck (hence why we posted here). P.S. This is an urgent listing so please get back to us if you think you can do the job!
ID del proyecto: 17984443

Información sobre el proyecto

13 propuestas
Proyecto remoto
Activo hace 5 años

¿Buscas ganar dinero?

Beneficios de presentar ofertas en Freelancer

Fija tu plazo y presupuesto
Cobra por tu trabajo
Describe tu propuesta
Es gratis registrarse y presentar ofertas en los trabajos
13 freelancers están ofertando un promedio de £64 GBP por este trabajo
Avatar del usuario
Hi! I can implement such task for you using python 3.6 and asyncio, but to be fair I prefer to use C++ whenever I need a maximum performance (even I we talk about networking). Feel free to msg me in chat. Sincerely, Ivan Stepin.
£23 GBP en 1 día
5,0 (7 comentarios)
5,5
5,5
Avatar del usuario
Hi, I have a knowledge using python with AWS lambda to do parallel requests. Per default, you can reach at maximum 5k requests at the same time using AWS lambda.
£18 GBP en 3 días
5,0 (6 comentarios)
4,3
4,3
Avatar del usuario
Hello! This resembles a load / stress testing scenario, which I've done a lot in the past prior to releasing numerous APIs to production. Python is too slow for this, GoLang would require a bit more programming than desired, however, there is a tool written in Scala on actors, called Gatling Gun that I've used to successfully fire up to 10 000 req/sec from a single machine, however, considering its architecture, it's well capable of doing even more. Changing the request data per request shouldn't be a problem as long there is either a source or a logical way of generating it. Proxy is a natural thing in HTTP, configuring requests to go through on also isn't an issue. Looking forward to hearing from you! Best regards, Tijana
£80 GBP en 2 días
5,0 (3 comentarios)
2,9
2,9
Avatar del usuario
Hello, I'm offering just a consultation mainly because I assume you have already tried the techniques I am going to suggest. If not I'm willing to help with the implementation as well. I'm focusing on python and C++ as I don't know golang. Obviously in python the simple approach is to use requests library which already allows to specify a proxy. Given the enormous amount of requests required the GIL will be a problem. Using an async implementation of requests or the multiprocessing package (true multithreading) or pypy could help around this problem. Likely python will still be too slow which is why an interface with c++ might be needed. I am most familiar with pybind11 which is amazingly fast compared to cpython. The multi-threading and sending would then be handled by the c++ code (which can use a c++ http library) and a high-level interface would be exposed to be usable from python. Likely even then os restrictions could be a problem. Working as root might clear some of them. I know ulimit can be used for restrictions like that. Additionally, the standard network driver in linux isn't incredibly fast (I have heard every package is copied around in memory about 20 times). Probably using a custom module/kernel can help. Hardware restrictions could likely only be resolved via using specialized hardware or using hardware in parallel.
£18 GBP en 1 día
5,0 (1 comentario)
2,4
2,4
Avatar del usuario
Hi, I've worked quite a bit with writing server code for highly parallel and concurrent servers with multiple protocols. I've worked with servers that bypass the host OS network stack, as well as from within the host stack as well. This has mostly been don in C and Python, with OS stacks being Linux and FreeBSD. From your post, I understand that you already have a solution, but it doesn't work well with 1K concurrent connections. Is that right? Can you let us know which stack this current solution is written in? We can try implement something with Twisted/Tornado or the one of the newer async frameworks in Python as Python/C is what I'm most familiar with, or if you prefer it in Go, we can do it in go as well. Do let us know if you need any more details from my side. Also please share more architectural information if you'd like to proceed. Thanks.
£111 GBP en 15 días
5,0 (1 comentario)
1,5
1,5
Avatar del usuario
Hello, I wrote a similar script to the one you are describing and it can be realized in a few lines of code using Python. However I do have a few questions about your project. What version of python do you want the implementation to be in? Do you want to send as many requests as possible? You mentioned varying payloads. Please eleborate a bit on this as well. In your listing you mentioned the possibility to assist you long term. As I am always for the lookout for more stable income, I would love to get some more information about that as well. With kind regards, Johannes Spieler
£18 GBP en 1 día
0,0 (0 comentarios)
0,0
0,0
Avatar del usuario
Hello! Your problem clearly. You can use any programming language. Anyway, eventually you work through the core of your operating system. Suggest you demo version of the program with parallel requests and responses to queries. Regards, Alex.
£23 GBP en 30 días
0,0 (0 comentarios)
0,0
0,0
Avatar del usuario
Hello, I think for this job we can use the language Golang to be faster than python. We can just start some goroutine to achieve all the requests that you want or use a lib for stress tests which I think can help us as well. I have some questions about the project that we can discuss.
£18 GBP en 10 días
0,0 (0 comentarios)
0,0
0,0
Avatar del usuario
Using Tensorflow Deep Learning library this can be achieved. Pure python multi threading or multiprocessing is limited. Take my bid in account for this challenge. Kind Regards, Dom
£333 GBP en 5 días
0,0 (0 comentarios)
0,0
0,0

Sobre este cliente

Bandera de UNITED KINGDOM
Luton, United Kingdom
4,9
7
Forma de pago verificada
Miembro desde ene 2, 2018

Verificación del cliente

¡Gracias! Te hemos enviado un enlace para reclamar tu crédito gratuito.
Algo salió mal al enviar tu correo electrónico. Por favor, intenta de nuevo.
Usuarios registrados Total de empleos publicados
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Cargando visualización previa
Permiso concedido para Geolocalización.
Tu sesión de acceso ha expirado y has sido desconectado. Por favor, inica sesión nuevamente.