Golang Stream Data to S3

Tablero de aclaración pública

• 
  Organizador del concurso

  • 4 años atrás

  it should have only one writer per file. if you generate a guid and append to the filename you'll know it is yours

  • 4 años atrás
  1. 

     yadavgajender087

     • 4 años atrás

     Usage of ./s3_uploader:
     -acl="bucket-owner-full-control": ACL for new object
     -bucket="": S3 bucket name (required)
     -chunk_size=50MB: multipart upload chunk size (bytes, understands standard suffixes like "KB", "MB", "MiB", etc.)
     -expected_size=0: expected input size (fail if out of bounds)
     -key="": S3 key name (required; use / notation for folders)
     -mime_type="binary/octet-stream": Content-type (MIME type)
     -region="us-west-2": AWS S3 region
     -retries=4: number of retry attempts per chunk upload
     -sse=false: use server side encryption

     • 4 años atrás
• 
  Organizador del concurso

  • 4 años atrás

  at least 10000qps using a max 8 core machine.

  • 4 años atrás
  1. 

     yadavgajender087

     • 4 años atrás

     is these one is right S3 has a maximum multipart count of 10000, therefore: total_input_size / chunk_size

     • 4 años atrás
• 

  yadavgajender087

  • 4 años atrás

  Stream to S3 from stdin using concurrent, multipart uploading.
  Intended for use with sources that stream data fairly slowly (like RDS dumps), such that getting the initial data is the dominant bottleneck. It is also useful to upload large files as quickly as possible using concurrent multipart uploading

  • 4 años atrás
• 
  Organizador del concurso

  • 4 años atrás

  Contest closes tomorrow. Looking forward to all the great submissions

  • 4 años atrás
• 

  ankurs13

  • 4 años atrás

  Is there any QPS expectation for this service (under what constraints)? Also, what should happen if the file corresponding to the message already exists in S3 (when the program starts)? Do we overwrite the file or append to it? Will there be multiple writers to the same log file? Do we need to handle that situation?

  • 4 años atrás

Mostrar más comentarios