After getting tired of no way to easily follow news of AWS product updates for only ones I actually use we built Broadside. You can set which services you care about, how often you want to be notified and it sends you a digest of the updates.
Jstanier - Great job, this looks really cool. Curious what icon set are you using? I saw you have similar style icons across a few of your projects, is there a standard library you are using or are you creating them yourself?
I /think/ I know what you're talking about, but just in case this gets derailed in a back-and-forth and lost, then feel free to contact me on Twitter @jstanier or my email is in my profile - we can chat more easily there.
At 500 million+ records, even a 0.01 second per lookup/update is going to be too slow at that scale even in parallel, unless the DB is able to process in parallel without lock contention extremely quickly.
Agreed that storage is cheap, unless it's in a live cluster. Right now I keep 100% of the data in Redshift then use window functions to unload the latest X per id. Keeping it in real time I haven't tried but right now the unloading of the full dataset of last X per id takes multiple days on a 6 node cluster.
The analysis at end of month is simply give me 100% of the data set chunked up by ID. All analysis is done outside of the system.
Flat files is my thinking right now with S3 and prefixes per partition, I'm not sure on file format, since one thing is with each day's data being able to process and update existing data quickly. Current thought is to load current day's data to Redshift -> unload sorted by id -> process concurrently. With multiple prefixes on S3 I won't hit the rate limits. My main worry is if read in, loading/parsing each file will take too long to be scalable at 250-500 million unique id's per day. I wanted to check here before going down that route to see if anyone had a different recommendation.
1. Hadoop / HDFS / Spark on an ephemeral cluster with disk snapshots
2. Group 1M ID's into a single file
3. If analysis is once a month, save daily then prep data right before analysis.
4. Consider using Cassandra database
5. Rent a big machine where the data can fit into memory
I've heard good things about them, for me right now the payroll takes ~30 minutes or so a month with end of quarter payroll taking a little longer to create the 941 forms.
We do the people at the same bank get free transfers, it's out of bank ones is ~5$ per transfer. The bank recommended using viewpost which charges at 50 cents/transfer but from their site they say they should not be used for payroll, and require the other side is registered on their network so it isn't just a routing+accounting numbers push ACH transfer.
It's great, I've been using one for ~2 years. I bought a powered adjustable one, but have only ever lowered it once when I had my ACL done about a year ago. Otherwise, I stand the entire day and find I can focus pretty well.
I also noticed I drink a bit more water and move around a little bit more since there isn't the "get up from sitting" process.
We're a small consulting shop based in Cambridge looking to add another person to our team. We work across a number of industries from advertising to FOREX to market research. We work with clients on an ongoing basis so much of our work is improving/adding to projects we previously built.
Most of our work is based in PHP (Symfony). We're looking for someone with a couple of years of experience to join us.
