We’ve worked with a lot of startups and see the the same anti-pattern over and
over again: the technical co-founder or CTO is the “server person”.
Whenever an application breaks or needs to be deployed, a database needs to be
configured, or it’s time to move clouds, it’s the CTO’s job.
The problem is that the CTO, the most valuable person on the engineering team,
is spending their time wrangling servers and clouds instead of managing their
team, building new features, fixing bugs, and hiring engineers.
This seems to happen because they’re the person who has worked on the project
the longest, knows the most, or was the one who originally set things up.
We built Flynn to change all that. Flynn simplifies ops by giving developers
self-serve access to infrastructure so they can deploy whatever they need to
whenever it’s ready. Flynn even runs databases for you.
Many Flynn users find they don’t need a full in-house ops team once they adopt
Flynn but they still need ops help. We take over ops for those companies, acting
as their 24/7/365 devops team. So we created Managed Flynn, which we also call
“Ops as a Service”.
We join their entire engineering team in a private Slack channel 24/7 to
instantly answer any questions or troubleshoot any problems that arise, whether
they’re related to Flynn or not.
We help them with application architecture, debugging, performance and database
tuning, and run of the mill questions that developers have. The result is that
the CTO is free to focus on building their product and team and we handle all
the day-to-day operations.
It also means the CTO can go on vacation without being “on call” in case
something breaks.
Flynn can run almost anywhere including all the major clouds and bare metal.
We’ve had a lot of luck helping our customers find free cloud credit and
seamlessly changing providers when it runs out. We’ve also had a lot of success
lowering cloud costs by switching to bare metal cloud providers like Packet.
We’re expanding our Ops-as-a-Service product to serve more customers. If you’re
an overworked co-founder or CTO and want to free yourself or your developers
from doing ops, let us know.
This is a security announcement that applies to all Flynn clusters that were
launched with the flynn install command on DigitalOcean or via SSH prior to
CLI version v20170108.1. No other cluster types are affected.
The firewall for clusters launched using these options was misconfigured,
leaving all ports defaulted to open instead of closed.
§
Impact
All clusters launched via flynn install with on DigitalOcean or via SSH that
have not had additional firewall rules manually applied are vulnerable to
unauthenticated remote code execution.
§
Remediation
There are two ways to fix the issue, manually adding the missing rules to an
existing cluster, or starting a new cluster using a fixed version of the
installer.
§
Start a new, fixed cluster
The flynn install command in versions v20170108.1 and later have this issue
fixed. Run flynn update and flynn version to confirm that your CLI version
is updated.
If you have apps and data that you’d like to keep, you can take a backup from an
existing cluster by running flynn cluster backup --file=flynn_backup.out and
restore it when starting a new cluster using flynn install. Make sure that you
destroy the old cluster after starting the new one.
§
Manually add firewall rules
The missing firewall rules can be added via SSH to the hosts in an existing cluster.
For single-host clusters
If the cluster only has one server, you can connect to it using SSH and add the
corrected firewall rules.
ssh -i ~/.flynn/installer/keys/flynn root@$CLUSTER_DOMAIN
The cluster domain typically starts with several alphanumeric characters and is
a subdomain of flynnhub.com, it can be obtained by looking at the output of
flynn cluster or git remote -v in a git repository that is configured to
deploy an app to the Flynn cluster.
Once connected to the server, run the following commands:
iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -i eth0 -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -i eth0 -j DROP
netfilter-persistent save
If the last command returns an error, the host is running Ubuntu 14.04, and the correct
command is:
/etc/init.d/iptables-persistent save
After running these commands, the correct firewall rules will be in place. See
the Testing section below for information on how to test this.
For multi-host clusters
If there is more than one host in the cluster, you will need to find the IPs of
each host and then connect to each and add the firewall rules.
You can get the list of IPs from the DigitalOcean dashboard, or by running
host against the cluster domain.
$ host aaa.flynnhub.com
aaa.flynnhub.com has address 127.0.0.1
aaa.flynnhub.com has address 127.0.0.3
aaa.flynnhub.com has address 127.0.0.5
The cluster domain typically starts with several alphanumeric characters and is
a subdomain of flynnhub.com, it can be obtained by looking at the output of
flynn cluster or git remote -v in a git repository that is configured to
deploy an app to the Flynn cluster.
Once you have the IP addresses of each host, connect to each of them and add the
firewall rules:
ssh -i ~/.flynn/installer/keys/flynn root@$HOST_IP
One rule will need to be added to allow access from each host in the cluster:
iptables -A INPUT -s 127.0.0.1 -j ACCEPT
iptables -A INPUT -s 127.0.0.3 -j ACCEPT
iptables -A INPUT -s 127.0.0.5 -j ACCEPT
Replace the 127.0.0.* IP addresses above with the IP address of each host in
the cluster, and make sure there is a rule added for each host IP.
After adding the rules for each host, add the general rules and save the
configuration:
iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -i eth0 -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -i eth0 -j DROP
netfilter-persistent save
If the last command returns an error, the host is running Ubuntu 14.04, and the correct
command is:
/etc/init.d/iptables-persistent save
After running these commands on each host, the correct firewall rules will be in
place. See the Testing section below for information on how to test this, and
make sure you test each host individually.
§
Testing
If the correct firewall rules are in place, only ports 80, 443, and 22 will be
open. You can quickly test if the rules are missing on a host by sending an HTTP
request to a host’s IP on a port that should be firewalled:
curl $FLYNN_HOST_IP:1111
If the firewall rule is missing, this will return a response, which means that
the issue has not been patched yet:
404 page not found
If the firewall rule is correctly configured, the request will timeout, this is
the output that you want to see:
curl: (7) Failed to connect to aaa.flynnhub.com port 1111: Operation timed out
A more extensive test can be done with nmap, this is the output that you
should see if the firewall is correctly configured:
$ sudo nmap -p- -sS $FLYNN_HOST_IP
...
Host is up (0.014s latency).
Not shown: 65532 filtered ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
Nmap done: 1 IP address (1 host up) scanned in 108.79 seconds
If other ports are listed as open, then the firewall is not correctly
configured.
§
Future Improvements
We are currently working on two projects that will help prevent
a misconfiguration like this from having the same impact in the future:
- The next version of the installer will have integration tests that ensure
that firewall rules are correctly applied to launched clusters.
- Internal cluster communications will be encrypted and authenticated so that
even if the firewall is misconfigured, exposing internal ports will have no
impact. We have a security
roadmap that covers this and
other issues.
§
Credits
Thanks to @KidkArolis for reporting this
issue.
If you encounter anything that you think may impact the security of Flynn,
please follow our disclosure process.
