Amazon Glacier: client is a Perl multithread/multipart downloading

image

Amazon Glacier


In short — Amazon Glacier is a service with a very attractive price of storage created to store files/backups. But the process of recovering files quite difficult and/or expensive. However, the service is quite suitable for secondary backup.
Read more about Glacier was already wrote on habré.

About the post


I want to share an Open Source client in Perl to synchronize a local directory with service Glacier also tell about some of the nuances of working with glacier and describe workflow of his work.

Functionality


So, mtglacier. Link on GitHub.
Software features:
the
    the
  • Protocol AWS is implemented independently, without using third party libraries. Self realization Amazon Signature Version 4 Signing Process calculate Hash Tree
    • the
  • Implemented Multipart Upload
    • the
  • multi-threaded upload/download/delete archives
    • the
  • Combining Mnohopocetneho and Multipart upload. I.e., three streams can upload part of the file A two — part file B, one can initiate upload of the file, etc.
    • the
  • Local journal file (opened for writing only in append mode). After all, to get a listing of files from Glacier you will need to wait 4 hours
    • the
  • the Ability to compare checksums of local files with the log
    • the
  • When downloading files, the ability to limit the number of files that will be downloaded at a time (needed because of the peculiarities of billing download archives)


project Goal


to Implement four things in one program (it's all there separately, but not together)

    the
  1. Implementation in Perl — I Think that the language/technology, which made the program too important to the end user/administrator. So it is better to have the choice of implementations in different languages
    2. the
  2. be Sure you plan to support Amazon S3
    3. the
  3. Multipart operation+multi-threaded operations
    multipart will help to avoid the situation, when you upload multiple gigabytes to a remote server and suddenly the connection breaks. Multithreading speeds up Sargasso, and speeds up loading heaps of small files or deleting large numbers of files
    4. the
  4. Own implementation of the Protocol is planned to make the code re-usability and to publish as a separate modules on CPAN


How it works



When you sync files to the service, mtglacier creates a magazine (text file) which contains all operations upload'and files: for each transaction, the name of the local file, upload time, Tree Nail received archive_id file.

When restoring files from Glacier to local disk, recovery data are taken from this log (because to get a listing of files on the glacier can only be delayed by four hours or more).

When deleting files from Glacier records are added about the removal in the log. When you re-sync to Glacier, processed only the files where there, according to the journal.

The recovery procedure is two-pass file:
    the
  1. create a task to download files present in the log, but not on the local disk
    2. the
  2. After waiting four hours you can re-run the command to download these files


Warnings


the
    the
  • Before use, please read prices on the website of Amazon
    • the
  • Examine the FAQ of Amazon

    • No, really, be sure to check the prices the

  • Before "play" with Glacier, make sure that you are able to delete all the files in the end. The fact that to remove files and is not empty vault not through Amazon Console
    • the
  • In multi-threaded work with the glacier you need to choose the optimal number of threads. Amazon returns the HTTP 408 Timeout in case of too slow injection in each specific thread (after this thread mtglacier pauses one second and retry, but no more than five times). So multi-threaded download may be slower than single-threaded


How to use


    the
  1. to Create a Vault in Amazon Console
    2. the
  2. to Create a glacier.cfg (specify the same region in which you created the vault)
    the 
    key=YOURKEY
secret=YOURSECRET
region=us-east-1

    3. the
  3. to Synchronize a local directory to Glacier. Use the option concurrency to specify the number of threads
    ./mtglacier.pl sync --config=glacier.cfg --from-dir /data/backup --to-vault=myvault --journal=journal.log --concurrency=3

    4. the
  4. you Can add files and sync again
    5. the
  5. to Check the file integrity (check only journal)
    ./mtglacier.pl check-local-hash --config=glacier.cfg --from-dir /data/backup --to-vault=myvault --journal=journal.log


    6. the
  6. Unable to remove some files from /data/backup
    7. the
  7. Create a job on data recovery. Use the option max-number-of-files to indicate the number of arhivov you want to restore. At the moment it is not recommended to specify a value more than a few dozen (still not loading more than one page with the list of current Jobs)
    ./mtglacier.pl restore --config=glacier.cfg --from-dir /data/backup --to-vault=myvault --journal=journal.log --max-number-of-files=10

    8. the
  8. Wait 4 hours or more
    9. the
  9. Restore deleted files
    ./mtglacier.pl restore-completed --config=glacier.cfg --from-dir /data/backup --to-vault=myvault --journal=journal.log

    10. the
  10. If a backup no longer needed, remove all files from Glacier.
    ./mtglacier.pl purge-vault --config=glacier.cfg --from-dir /data/backup --to-vault=myvault --journal=journal.log



Implementation of


the
    the
  • Operation with HTTP/HTTPS implemented through LWP::UserAgent
    • the
  • interaction with the API of Amazon written from scratch, only used the module Digest::SHA (Core module)
    • the
  • Independent implementation of Amazon Tree Hash (it's their own algorithm for checksum calculation, found similar among TTH algorithms, please correct if wrong)
    • the
  • implemented multi-Threading using fork processes
    • the
  • Processing an abort processes with signals

    • the interaction between the processes using unnamed pipes the

  • job queues — your OOP objects, like FSM

not yet enough


The main thing for the first beta was — stable working (it's not alpha) version, ready by the end of the week, so a lot of that is not enough.
Definitely will:
the
    the
  • Adjustable chunk size
    • the
  • -the multipart upload
    • the
  • the topic for SNS notifications in the config
    • the
  • integration with the outside world for receiving a signal via SNS notification
    • the
  • Internal refactoring
    • the
  • Will be published unit tests when you take them in order
    • the
  • Will be another testsuite, probably a server emulator glacier
    • the
  • of Course is a production-ready version (not Beta)
Article based on information from habrahabr.ru

Комментарии

Отправить комментарий

Популярные сообщения из этого блога

Monitoring PostgreSQL + php-fpm + nginx + disk using Zabbix

Изображение
a Lot of information in the network according to Zabbix, and a lot of bespoke templates, I want to introduce to the audience his modification. Zabbix is a very convenient and flexible monitoring tool. Want a hundred monitor, I want a thousand stations, and do not want to — watch a single server, take the cream in all sections. I would not mind to give to github, if anyone collects similar. It so happened that we decided to put on the hosting database wrapper php-fpm+nginx. As the database is postgres. Thoughts to collect data on the machine was before the purchase of hosting is needed, this is useful! Magic kick up the backside to the implementation of the system gave the brakes a hard drive on our VDS stations at the beginning of the script every minute, put the time and measured by the speed in the file, and then build graphs in Excel to compare to as it was/became, take quantitative statistics. And this is just one option! And suddenly the fault is not VDS, and our a
Далее...

Templates ESKD and GOST 7.32 for Lyx 1.6.x

Изображение
for more than six months I use a word processor Lyx. Built-in templates is more than enough to write lecture notes, prepare for seminars and to do simple reports for laboratory work. But last semester I had to make two course of the project in accordance with the requirements ESKD. I have begun work in Lyx, but without special templates of documents, my reports in the editor looked like this: A little work, a little reading of the document "Customizing LyX: Features for the Advanced User" and it turned out where the beam: You can use immediately. If you have experience of working in Lyx, problems will not arise. Just in case the attached sample documents. Description of how to make such templates are not yet doing. Someone will ask — will tell. ESKD Screenshot menu select unit type: Group "sections and subsections" appeared due to the fact that the document can be sections with sections without subsections — for their handling of para
Далее...

Customize your Google

Изображение
After writing this post, I was interested in the search parameters which you can use for convenient operation. Given this issue, in Runet I'm more or less full information is not found. But! It turns out Google for a long time and for all shown . Let's start simple — www.google.com/search?q=Запрос this is the simplest query, in which all options are disabled or parameters are used which are stored in your cookies. Next I will talk about the most interesting and used parameters. If you want to know more, you here . Even though it is called the search parameter, I will use the word token. So any of the marks are for the test, insert the end of the line (if your line is of the same kind as just above). Marker search by country &cr=countryRU Putting this token, we get the results from the specified country. List of acronyms used. Example Asus Finland UPDATE (thanks to Soutlan noticed the error) Token language results &lr
Далее...