I've spent the last week or so working on getting a demo server running for sqlChain. This is just a first step on the way to a more functional and useful explorer site.

I started out on Amazon EC2 with an m4.large instance. It's familiar and easy to start a server and get to it. I created a 50GB EBS data volume and attached that. Trying to save money I mistakenly chose the "Magnetic" type. First time I tried that, and little did I know that it isn't the same as "Cold HDD". I thought it cost 2.5 cents/GB/mo. but it was actually 5 cents. The real kicker is this EBS type charges for I/O and 42 million DB accesses later I realized this. Ehh, nuked that volume and thought more about using a VPS where resources aren't pay-as-you-go.

After a brief scan of LowEndBox.com I found Hudson Valley had a 4-vCPU-4GB-300GB offering for $5/mo. and takes Bitcoin. That sounds great, and even though it's not SSD storage, I thought it would do for testing. I sync'd up to block 389,000 before the IOPS of hard disk storage became painful and started looking for a temporary high end solution. Next up - Vultr.com because they also take Bitcoin. They require you to initially fund with Paypal or credit card. I guess that's to link your real world identity to the account. No worries. I added $5 from Paypal and got a $20 bonus credit to play with. Later I found out with some Google-fu I could have got $50 free credit.

The Hudson Valley VPS is your traditional pay-per-month but Vultr is more a cross between a VPS and Amazon cloud computing. You pay-per-use but not as granular and not as flexible with many niceties missing. I needed something with SSD and more speed. I started a quick test on the 1-vCPU-1GB VPS and noticed the storage speed is not even close to real SSD specs. I guess they use the SSD speed to cram more users in for a cheap price. So the next step was a "hybrid" server - they have a 2-CPU-8GB-120GB-SSD for 8 cents/hour. I later tried the 4-CPU offering as well but it gave little more oompf. It did prompt me to rework some of the threading in sqlChain and try to get more connections open to MySQL. So the 2-CPU "dedicated" option was the best so far. No extra charges for bandwidth and IOPS but with more access to raw CPU power. I have to say being able to test out different specs for a few hours, without bothering support or much hassle, is very nice.

I installed sqlChain as per my install guide using the PyPi package and was up and syncing within minutes. Well, almost. Turns out they mess around with the normal Ubuntu install, missing some things and opening up root logins. So first I had to do a bit of server admin to get a customary environment and then zip-zap, up and sqlChaining.

I rsync'd about 75 GB of data from Hudson Valley over to this new system, and continued on. Until my next lesson in hard knocks: don't test new code on a half sync'd database because you may have to start again. I had written the new threading code and local testing showed it should be more than double the speed. The hitch? It actually wasn't working correctly and threw away most of the outputs from about 4 million txs before I noticed. Fix, fix, fix. Tedious sql maneuvering to figure out what txs are toast and delete part of the database, then restart. On it goes... but it does run faster. Tomorrow some numbers...