Great stuff, wonder how he convinced the team to drop the "for free" academic HPC/Slurm mess in favor of costly cloud computing. That is my biggest hurdle working at a university right now.
mhm... having skimmed through the article, the only reason (imho) is not wanting to wait for hpc/slurm to schedule your jobs... all the other stuff, could have been done with university resources. They usually offer some shared drive space and are often forthcoming in supporting special needs (at least from my experience).
Having used both AWS Spot Instances for compute jobs, as well as hpc resources (slurm), the amount of work to getting it run is comparable... Except that the aws stuff quickly gets expensive when needing thousands of cpu hours per "experiment", and money is usually tight at universities ;-). Some ansible scripts automate basically everything away for myself. But using the tool you know is probably not always a bad idea.
Thanks! AWS was costly compared to the free HPC cluster but it was sub $1000 per month and our bottlenecks were person-hours rather than money. We had a lot of work to do. As some have mentioned there wasn't a strong technical argument for switching to AWS, it was more about productivity with those tools. I'd built up a bit of credibility with the team at that point and was like "I can make this better" and my boss was said "ok cool". As I mentioned in the article the team was very open to try new things.
I can see an alternate timeline where I set up Buildkite in AWS to run Slurm jobs on the HPC cluster, rather than using EC2 spot instances. Using Buildkite + S3 were probably the more important infra changes.
A lot of companies are donating computing power for COVID-19 research, so it's possible they have a deal of some kind worked out with Amazon.
> That is my biggest hurdle working at a university right now.
We took the opposite approach and assembled our own equipment, which we self host and manage. The cost savings we're significant, and what we lose doing admin and maintenance is saved by having the flexibility and freedom to code and run what we want, when we want.
If you've got the data center for it. Otherwise it's great until you have to upgrade your AC, and punch a hole in the side of the building to get the new one brought in on a crane :-)
I built and ran a 1mw data center for about 7 years, it can be a headache but you can be ahead of the curve if you buy a DC that’s being vacated by someone moving to the cloud.
I paid 400k to buy about 4m worth of equipment from the previous tenant. UPS, STS, busbar distribution, 3n paralleling generators and 20,000 gallons of diesel.
> wonder how he convinced the team to drop the "for free" academic HPC/Slurm
Probably because they are working with publicly available data so they don't need to worry about HIPAA/restricted data compliance. One of the main reasons for using the HPC/Slurm cluster is that IT (or the data compliance office) guarantees compliance with restricted data.
I had a case where using the "for free" academic HPC cluster was so inconvenient that we used only in very few cases when we needed very large amounts of compute, and for any smaller tasks using the commercial cloud solutions saved enough time/salaries so that they were worthwhile even purely from cost perspective. Also, for academic purposes it was possible to get quite large amounts of cloud computing for free, so most users never bothered to even try to approach the academic HPC environment as it had no advantages whatsoever.
DTU Bioengineering | Faculty position | ONSITE | Full-time
We are looking to fill a Data Science faculty position (different levels possible: Assistant Prof/Researcher or Associate Professor/Senior Researcher) in the Department of Biotechnology and Biomedicine at the Technical University of Denmark.
Do you have a track record in deploying Machine Learning models to support data-driven decision making in complex engineering projects? Do you have experience with Data Science and Data Engineering in the context of Biotechnology and/or you are convinced that Biology is the next field to be disrupted by Machine Learning? Do you enjoy enabling scientific breakthroughs through your expertise and participating in collaborative research projects? Do you have an interest in robotics, Internet of Things, Digital Twins and other Industry 4.0 elements and how they can be applied to bioengineering? Do you appreciate being consulted on experimental design and data management? Please apply if you resonate with some of these statements.
Novo Nordisk Foundation Center for Biosustainability (DTU Biosustain) | Full Stack Engineer | Copenhagen Denmark | ONSITE | VISA
The project
Modern genome engineering technologies have enabled the rapid and precise editing of DNA, causing a paradigm shift in biotechnology: with the focus shifted from reading to writing genetic information, which of the three billion DNA letters in a human cell need to be rewritten in order to prevent disease, live longer, or encode entirely novel features?
The project we are hiring you for is about building a web-based application for computational aided design of cells. Communicating with biological knowledge bases and mathematical models (mechanistic and AI-based), the platform will guide engineers in writing the right DNA based on the entire computable knowledge available for a cell.
The job
We are looking for a full-stack developer. You will be dealing with biological concepts every day, discovering the best user experience for interacting with DNA and its encoded information. Deep understanding of biology is not a requirement though, most of the developers in the institute were able to gain the necessary knowledge after joining our team.
Our technology stack at the moment is:
- Angular, Angular Material, Typescript
- Python3.6 for the backend services (Flask, celery, etc.) and core scientific libraries
Novo Nordisk Foundation Center for Biosustainability (DTU Biosustain) | Full Stack Engineer | Copenhagen Denmark | ONSITE | VISA
The Center for Biosustainability at the Technical University of Denmark is an international research center developing next generation cell factories and bioprocesses for sustainable production of high-value chemical compounds as well as protein-based products. The center uses advanced metabolic engineering techniques and computational biology ensuring efficient and cost-effective design and construction processes.
Engineering a living cell is an incredibly hard task. Mathematical and computational models are being developed in order to make computer-aided design of biology possible. The project we are hiring you for is about building a web platform to provide easy access to metabolic models so the development of new strains becomes faster and cheaper. The platform already has the capabilities of modifying cells right in your browser window and constraining the models with real experimental data in order to gain insights about cellular metabolism. You can get more information about the project at http://dd-decaf.eu/ and take a look at the platform itself at https://app.dd-decaf.eu.
The Job:
We are looking for a full-stack developer with a strong focus on the front-end. You will be dealing with biological concepts every day, discovering the best ways to operate with genes and biochemical reactions. Deep understanding of biology is not a requirement though, most of the developers in the institute were able to gain the necessary knowledge after joining our team. Our technology stack at the moment is
* Angular, Angular Material, Typescript
* Python3.6 for the backend services (Flask) and core scientific libraries
* Postgres, MongoDB, Redis, Neo4j
* Docker
NNF Center for Biosustainability | Full Stack Engineer | Copenhagen Denmark | ONSITE | VISA
Full-stack Developer for Synthetic Biology Platform
The focus of The Novo Nordisk Foundation Center for Biosustainability is synthetic biology and strain design. Living cells are being genetically modified in order to produce chemicals in a sustainable way. The list of things that can be made by engineered strains includes biofuels, bioplastics, drugs, food supplements and much more.
Engineering a living cell is an incredibly hard task. Mathematical and computational models are being developed in order to make computer-aided design of biology possible. The project we are hiring you for is about building a web platform to provide easy access to metabolic models so the development of new strains becomes faster and cheaper. The platform already has the capabilities of modifying cells right in your browser window and constraining the models with real experimental data in order to gain insights about cell metabolism. You can get more information about the project at http://dd-decaf.eu/.
We are looking for a full-stack developer with a strong focus on the frontend. You will be dealing with biological concepts every day, discovering the best ways to operate with genes and biochemical reactions. Deep understanding of biology is not a requirement though, most of the developers in the institute were able to gain the necessary knowledge after joining the team. Our technology stack at the moment is
* AngularJS (ongoing migration to Angular 5), Angular Material, Typescript
* Python3.6 for the backend services (aiohttp framework) and core scientific libraries
* Postgres, MongoDB, Redis, Neo4j
* Docker
Full-stack Web Developer for Synthetic Biology Design Platform
We're looking for a software engineer to help build a web-based software platform for data-driven design of microbial cells and communities. DTU Biosustain conducts research on metabolic engineering and synthetic biology of microbial and mammalian cell factories for chemical and therapeutic protein production.
With advances in synthetic biology, genomes can now be edited at unprecedented speed and fidelity allowing making multiple changes in the same genome at the same time. This increases the need for computational tools to design cells and communities of cells analogous to the tools used in Computer Aided Design (CAD) of cars, buildings and other man-made objects.
DTU Biosustain | Copenhagen Area, Denmark | Onsite | Full-time | VISA
Full-stack Web Developer for Synthetic Biology Design Platform
With advances in synthetic biology, genomes can now be edited at unprecedented speed and fidelity allowing making multiple changes in the same genome at the same time. This increases the need for computational tools to design cells and communities of cells analogous to the tools used in Computer Aided Design (CAD) of cars, buildings and other man-made objects. The European Commission has awarded 6.3 million Euros to a four-year collaborative project to build a CAD tool for data-driven design of cells and microbial communities. Applications will range from human health to sustainable production of chemicals.
Our web stack currently, in no specific order: AngularJS (soon Angular2), Angular Material, Typescript, Python 3, PostgreSQL, Docker, gRPC & protobuf. The API is written in Python and our scientists write most of their algorithms in Python. We currently use AngularJS for the front-end, however this is not set in stone. As visible on our GitHub org (https://github.com/biosustain), we are very dedicated towards releasing most of our tools as open source software.
We will help with getting a VISA, as well as with administrative issues related to relocation. Our working language is English. We are a very diverse workplace; 70% of our staff have an international background. The salary depends on qualifications and needs to be agreed by your union representative -- so I cannot give an official range -- but a rough personal estimate is 420k-500k DKK.
So if this sparked your interest, come and join a diverse team of scientists and engineers to work on something that matters.
Front-end Web Developer for Synthetic Biology Design Platform
With advances in synthetic biology, genomes can now be edited at unprecedented speed and fidelity allowing making multiple changes in the same genome at the same time. This increases the need for computational tools to design cells and communities of cells analogous to the tools used in Computer Aided Design (CAD) of cars, buildings and other man-made objects. The European Commission has awarded 6.3 million Euros to a four-year collaborative project to build a CAD tool for data-driven design of cells and microbial communities. Applications will range from human health to sustainable production of chemicals.
Our web stack currently, in no specific order: AngularJS, ECMAScript 6, Redis, Python 3, Flask, PostgreSQL 9.4, Socket.IO, Docker. The application is very front-end centric so most of the work will always be done in JavaScript. The API is written in Python and our scientists write most of their algorithms in Python. We currently use AngularJS for the front-end, however this is not set in stone.
As visible on our GitHub org (https://github.com/biosustain), we are very dedicated towards releasing most of our tools as open source software.
We will help with getting a VISA, as well as with administrative issues related to relocation. Our working language is English. We are a very diverse workplace; 70% of our staff have an international background.
The salary depends on qualifications and needs to be agreed by your union representative -- so I cannot give an official range -- but a rough personal estimate is DKK 450k-600k.
So if this sparked your interest, come and join a diverse team of scientists and engineers to work on something that matters.
Denmark, Copenhagen | DTU Biosustain, Technical University of Denmark | ONSITE | VISA
The Novo Nordisk Foundation Center for Biosustainability (DTU Biosustain) at the technical university of Denmark (DTU) is seeking to hire two highly motivated and skilled software engineers to help build a web-based software platform for data-driven design of microbial cells and communities.
The Center conducts research on metabolic engineering and synthetic biology of microbial and mammalian cell factories for chemical and therapeutic protein production. The position will be within EU funded project to develop new bioinformatics approaches that leverage system-level data for biotechnological applications.
With advances in synthetic biology, genomes can now be edited at unprecedented speed and fidelity allowing making multiple changes in the same genome at the same time. This increases the need for computational tools to design cells and communities of cells analogous to the tools used in Computer Aided Design (CAD) of cars, buildings and other man-made objects. The European Commission has awarded 6.3 million Euros to a four-year collaborative project to build such a CAD tool for data-driven design of cells and microbial communities. Applications will range from human health to sustainable production of chemicals.
As a successful applicant, you will be part of a diverse team of scientists and engineers from leading academic institutions and innovative companies that will work towards this common goal.
For more details of the two positions, and to apply, please visit:
