Has anyone seen max (p100) client latencies of 300 to 400ms but totally normal p99? We see this across almost all our redis clusters on elasticache and have no idea why. CPU usage is tiny. Slowlog shows nothing.
I would guess your problem is probably scheduler based. The default(ish) Linux scheduler operates in 100ms increments, the first use of a client takes 3-4 round-trips. TCP opens, block, request is sent, the client blocks on write, the client attempts to read and blocks on read. If CPU usage is high momentarily, each of these yields to another process and your client isn't scheduled for another 100ms
Are you evicting or deleting large sets (or lists or sorted sets)? We use a Django ORM caching library that adds each resultset's cache key to a set of keys to invalidate when that table is updated – at which point it issues `DEL <set key>` and if that set has grown to hundreds of thousands – or millions! – of keys the main Redis process will block completely for as long as it takes to loop through and evict them.
Is the memory full and evicting? Or do you have a large db with lots of keys with ttls? Redis does a bunch of maintenance stuff on the same thread iirc in the background but not really
What's worse is the documentation straight up lies. It states you can perform a major version upgrade by resorting a snapshot and selecting a higher version. I mean it's still not ideal except if you do try this you'll find the option doesn't actually exist - either via the console or API/CLI!
Yes, please coordinate with kirsten@gitlab.com for a 25 minute Google Hangout with me, please link to this comment in your email to her. If the advice is generic enough we might work with you to publish it https://about.gitlab.com/handbook/people-operations/ceo-pref... (you have a final say in what gets published). This process was also used to produce the OP.
I got curious too; in the Berkeleyside article about it, they say "right now, posts come from a combination of community users, Wildfire staffers and local news partners, including Berkeleyside. To verify information, and decide whether or not to push out an alert to the community, the team checks posts against what is released by authorities and the local media, too."
If possible, private would be better for us. I'd be happy to make literally any 25' slot you have over the next 2 days. This is really generous of you, and we really appreciate it. Thanks!
No problem, please email kirsten@gitlab.com to coordinate a time, please link to this comment, if you want it the next 2 days my preference is Sunday 4pm Pacific but I'm not sure that works for you. I propose we do https://about.gitlab.com/handbook/people-operations/ceo-pref... (you get to redact the notes)
They mentioned using hadoop for file storage - perhaps they are just using HDFS and not MapReduce.
Otherwise, Spark is relatively new, so they might have some older infra/jobs in Hadoop.
Storm and Spark streaming work a little differently (real-time streaming vs "micro-batching) and apparently have different use cases, but I'm not totally sure what the practical difference are here either..
Skilledup Academy (New York)- Senior Ruby Developer
TL;DR:
We’re looking for a Senior Ruby Developer to help us build a mentored online education platform to get people their first job in a new career. You are a coder, a system designer, and a mentor. More than that, the Senior Ruby Developer is a crucial, respected contributor to a startup team that is playing to win. Our code is pristine. We are nice.
We are looking for 2+ years of professional rails experience. Our website is at academy.skilledup.com.
Our Beliefs:
§ Education changes the world: We're an education company at heart. We care about the growth of our learners and our employees. We have investment days on Fridays to research the competition, learn new skills, and do things that get us closer to our vision.
§ Sustainable Code: We practice BDD. Our code is pristine and we're proud of the work we produce. We have over 95% test coverage, but we focus on writing the right kinds of tests.
§ Lean: We don't build things we don't need. We iterate quickly on what we learn.
§ Cross-Functional Teams: Our team is composed of generalists. There is no "wall" to throw things over. We aren’t fans of the "assembly line" model of development.
§ Flat Organization: We believe good ideas come from anyone in the crowd. In that sense, we are a flat organization.
§ Design First: We really care about UX and design, and bring 3 to 5 users in every week for user testing so that we stay close to our users. We first design wide, then design deep.
Tools include: Ruby on Rails, Postgres, Heroku, RSpec, Factory Girl, CoffeeScript, Konacha & Mocha, Trello, GitHub, git rebase -i, CodeClimate, Gemnasium, and CircleCI.
SkilledUp Academy is changing the way that people learn skills online. Over 44% of college graduates work in jobs that don't require a college degree, but employers still can't find quality talent. We introduce the human element into real-world, skills-based online education. Students complete rigorous curriculums created by our partner companies that focus on teaching the skills that employers need. Our industry-based mentors ensure that students receive the support that they need to learn.
Our Story
We were born out of two acquired startups and still retain the culture and practices of a lean / agile startup, but have all the resources and benefits of a public company (our parent company, Apollo Education Group, made over 3.5 billion in revenue last year). We’re a young company looking for people looking to help shape our culture.
Our Team
We’re a diverse team made of:
- A Bonnaroo Guitarist
- The world’s preeminent scholar on Toilet Based Social Networks
- A philosopher king with a sweet spot for ice cream
Our engineers write pristine, well-tested code. We pride ourselves in developer happiness. SkilledUp uses a pull request based workflow, continuous integration. We think Ruby on Rails, Trello, Github, and Heroku are the best tools. We have continuous investment days, where we only focus on learning new skills.
If you are interested in learning more and want to talk to an engineer, send an email to jon@skilledup.com
Senior Ruby/Javascript Engineer - SkilledUp Academy
New York, NY
Our Mission
SkilledUp Academy is changing the way that people learn skills online. Over 44% of college graduates work in jobs that don't require a college degree, but employers still can't find quality talent. We introduce the human element into real-world, skills-based online education. Students complete rigorous curriculums created by our partner companies that focus on teaching the skills that employers need. Our industry-based mentors ensure that students receive the support that they need to learn.
Our Story
We were born out of two acquired startups and still retain the culture and practices of a lean / agile startup, but have all the resources and benefits of a public company (our parent company, Apollo Education Group, made over 3.5 billion in revenue last year). We’re a young company looking for people looking to help shape our culture.
Our Team
We’re a diverse team made of:
- A Bonnaroo Guitarist
- The world’s preeminent scholar on Toilet Based Social Networks
- A philosopher king with a sweet spot for ice cream
Our engineers write pristine, well-tested code. We pride ourselves in developer happiness. SkilledUp uses a pull request based workflow, continuous integration. We think Ruby on Rails, Trello, Github, and Heroku are the best tools. We have continuous investment days, where we only focus on learning new skills.
If you are interested in learning more and want to talk to an engineer, send an email to jon@skilledup.com
