RSS

Startup CTOs Shouldn't Do Ops

We’ve worked with a lot of startups and see the the same anti-pattern over and over again: the technical co-founder or CTO is the “server person”.

Whenever an application breaks or needs to be deployed, a database needs to be configured, or it’s time to move clouds, it’s the CTO’s job.

The problem is that the CTO, the most valuable person on the engineering team, is spending their time wrangling servers and clouds instead of managing their team, building new features, fixing bugs, and hiring engineers.

This seems to happen because they’re the person who has worked on the project the longest, knows the most, or was the one who originally set things up.

We built Flynn to change all that. Flynn simplifies ops by giving developers self-serve access to infrastructure so they can deploy whatever they need to whenever it’s ready. Flynn even runs databases for you.

Many Flynn users find they don’t need a full in-house ops team once they adopt Flynn but they still need ops help. We take over ops for those companies, acting as their 24/7/365 devops team. So we created Managed Flynn, which we also call “Ops as a Service”.

We join their entire engineering team in a private Slack channel 24/7 to instantly answer any questions or troubleshoot any problems that arise, whether they’re related to Flynn or not.

We help them with application architecture, debugging, performance and database tuning, and run of the mill questions that developers have. The result is that the CTO is free to focus on building their product and team and we handle all the day-to-day operations.

It also means the CTO can go on vacation without being “on call” in case something breaks.

Flynn can run almost anywhere including all the major clouds and bare metal. We’ve had a lot of luck helping our customers find free cloud credit and seamlessly changing providers when it runs out. We’ve also had a lot of success lowering cloud costs by switching to bare metal cloud providers like Packet.

We’re expanding our Ops-as-a-Service product to serve more customers. If you’re an overworked co-founder or CTO and want to free yourself or your developers from doing ops, let us know.

January Stable Release, NYC Meetup

Our January release of Flynn includes a variety of bugfixes and minor improvements. The changelog has the details and there are docs on how to update.

§ NYC Meetup

We’ll be at the Downtown NYC Tech Meetup talking about Flynn on Thursday, January 19th. Please RSVP, space is limited!

§ Security Announcement

If have a Flynn cluster on DigitalOcean that was deployed using flynn install, please read this post about an important security issue that affects you.

§ Stay in Touch

§ How You Can Help

Security Announcement: DigitalOcean installer firewall misconfiguration

This is a security announcement that applies to all Flynn clusters that were launched with the flynn install command on DigitalOcean or via SSH prior to CLI version v20170108.1. No other cluster types are affected.

The firewall for clusters launched using these options was misconfigured, leaving all ports defaulted to open instead of closed.

§ Impact

All clusters launched via flynn install with on DigitalOcean or via SSH that have not had additional firewall rules manually applied are vulnerable to unauthenticated remote code execution.

§ Remediation

There are two ways to fix the issue, manually adding the missing rules to an existing cluster, or starting a new cluster using a fixed version of the installer.

§ Start a new, fixed cluster

The flynn install command in versions v20170108.1 and later have this issue fixed. Run flynn update and flynn version to confirm that your CLI version is updated.

If you have apps and data that you’d like to keep, you can take a backup from an existing cluster by running flynn cluster backup --file=flynn_backup.out and restore it when starting a new cluster using flynn install. Make sure that you destroy the old cluster after starting the new one.

§ Manually add firewall rules

The missing firewall rules can be added via SSH to the hosts in an existing cluster.

For single-host clusters

If the cluster only has one server, you can connect to it using SSH and add the corrected firewall rules.

ssh -i ~/.flynn/installer/keys/flynn root@$CLUSTER_DOMAIN

The cluster domain typically starts with several alphanumeric characters and is a subdomain of flynnhub.com, it can be obtained by looking at the output of flynn cluster or git remote -v in a git repository that is configured to deploy an app to the Flynn cluster.

Once connected to the server, run the following commands:

iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -i eth0 -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -i eth0 -j DROP
netfilter-persistent save

If the last command returns an error, the host is running Ubuntu 14.04, and the correct command is:

/etc/init.d/iptables-persistent save

After running these commands, the correct firewall rules will be in place. See the Testing section below for information on how to test this.

For multi-host clusters

If there is more than one host in the cluster, you will need to find the IPs of each host and then connect to each and add the firewall rules.

You can get the list of IPs from the DigitalOcean dashboard, or by running host against the cluster domain.

$ host aaa.flynnhub.com
aaa.flynnhub.com has address 127.0.0.1
aaa.flynnhub.com has address 127.0.0.3
aaa.flynnhub.com has address 127.0.0.5

The cluster domain typically starts with several alphanumeric characters and is a subdomain of flynnhub.com, it can be obtained by looking at the output of flynn cluster or git remote -v in a git repository that is configured to deploy an app to the Flynn cluster.

Once you have the IP addresses of each host, connect to each of them and add the firewall rules:

ssh -i ~/.flynn/installer/keys/flynn root@$HOST_IP

One rule will need to be added to allow access from each host in the cluster:

iptables -A INPUT -s 127.0.0.1 -j ACCEPT
iptables -A INPUT -s 127.0.0.3 -j ACCEPT
iptables -A INPUT -s 127.0.0.5 -j ACCEPT

Replace the 127.0.0.* IP addresses above with the IP address of each host in the cluster, and make sure there is a rule added for each host IP.

After adding the rules for each host, add the general rules and save the configuration:

iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -i eth0 -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -i eth0 -j DROP
netfilter-persistent save

If the last command returns an error, the host is running Ubuntu 14.04, and the correct command is:

/etc/init.d/iptables-persistent save

After running these commands on each host, the correct firewall rules will be in place. See the Testing section below for information on how to test this, and make sure you test each host individually.

§ Testing

If the correct firewall rules are in place, only ports 80, 443, and 22 will be open. You can quickly test if the rules are missing on a host by sending an HTTP request to a host’s IP on a port that should be firewalled:

curl $FLYNN_HOST_IP:1111

If the firewall rule is missing, this will return a response, which means that the issue has not been patched yet:

404 page not found

If the firewall rule is correctly configured, the request will timeout, this is the output that you want to see:

curl: (7) Failed to connect to aaa.flynnhub.com port 1111: Operation timed out

A more extensive test can be done with nmap, this is the output that you should see if the firewall is correctly configured:

$ sudo nmap -p- -sS $FLYNN_HOST_IP
...
Host is up (0.014s latency).
Not shown: 65532 filtered ports
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 108.79 seconds

If other ports are listed as open, then the firewall is not correctly configured.

§ Future Improvements

We are currently working on two projects that will help prevent a misconfiguration like this from having the same impact in the future:

  1. The next version of the installer will have integration tests that ensure that firewall rules are correctly applied to launched clusters.
  2. Internal cluster communications will be encrypted and authenticated so that even if the firewall is misconfigured, exposing internal ports will have no impact. We have a security roadmap that covers this and other issues.

§ Credits

Thanks to @KidkArolis for reporting this issue.

If you encounter anything that you think may impact the security of Flynn, please follow our disclosure process.

December Stable Release

Flynn’s December stable release is here! It includes support for log streaming to remote syslog servers, expanded Ubuntu 16.04 support, and various stability improvements. The changelog has the details and there are docs on how to update.

§ Our Vision

Our vision is to free everyone from operating software. Read more.

§ Managed Flynn

Deploying Flynn is easy, but we can make it even easier by doing it for you! You get a private Slack channel, and we act as your ops team, helping you take advantage of Flynn and optimize your apps.

§ Stay in Touch

§ How You Can Help

November Stable Release

The November stable release is here! It includes many bugfixes and a completely rewritten container image subsystem that significantly improves performance for common actions. The changelog has the details and there are docs on how to update.

§ More Documentation

We’re writing more docs and tutorials, tell us what you want to see using this form!

§ Deploying from Travis CI

Evan Tahler wrote a great guide on doing continuous deployment from Flynn to Travis CI.

§ Managed Flynn

Deploying a Flynn cluster is easy, but we can make it even easier by doing it for you! You get a private Slack channel, and we act as your ops team, helping you take advantage of Flynn and optimize your apps.

§ Stay in Touch

§ How You Can Help

 RSS



Mailing List