Launching a weekly newsletter

Tomorrow we are launching a weekly newsletter for clients and developers on Moonlight. We hope it can decrease email noise, and better let you know what is most relevant and helpful to you on Moonlight!

Benefits for everybody:

  • A digest of your Moonlight action items
  • See latest community posts and updates

Benefits for developers:

  • Review latest job matches
  • Add latest skills to your profile for better matching

Benefits for companies:

  • Review new and recommended developer matches
  • Get status updates about open job posts

This is an experimental system that we built in-house. Let us know your thoughts and feedback, plus any bugs you find!

One-click unsubscribe

It's now easier to manage the emails that you receive from Moonlight. Marketing emails now include a one-click unsubscribe link in their footer.

Manage your email subscriptions, and see the type of emails that Moonlight sends on the settings page.

Cleaner email templates

We've updated our standard email template! It looks cleaner and more consistent with our latest branding.

If the layout happens to render incorrectly in your email client, please let us know so that we can fix it.

Old:

Screen Shot 2019-02-26 at 3.25.57 PM.png

New:

Screen Shot 2019-02-26 at 3.26.34 PM.png

Referrals: Invite and earn $500

Moonlight's professional community is made up of thousands of developers, designers, product managers, and founders. We value our existing members and want to fairly incentivize you for growing the network. So, today we are updating and expanding our referral program!

💼 Clients can now participate in the referral program

Clients of Moonlight can now set up a referral link and get paid for referring new users. Just set up payments to activate your link.

💰 Increasing the client referral payout to $500

Get $500 when a new client signs up through your referral link and becomes a paying member. We'll send cash to your connected Stripe account automatically!

👷‍♀️New reward program for developer referrals

Get Moonlight swag for every developer you refer! We'll send you a gift from the remote work store when new developers sign up through your link and are accepted to the community.

➡️ Get started

Get your personal links to share on the referrals page of the Moonlight website!

Get your referral link

Moonlight Referral Program

Q&A

What does a client have to spend to trigger a payout?

They need to spend $1000 on memberships or job posts. With our current pricing, this means that any single purchase will trigger a payout.

Can I send an email introduction?

Yes! Just send an introduction email to founders@moonlightwork.com, and we will credit you for resulting signups. Copy our recommended intro message on the referrals page!

Outage post-mortem

From January 18 to January 22, Moonlight's website and API had intermittent outages. These started as occasional API errors but culminated in periods of downtime yesterday. The problems have been resolved, and the application has returned to normal.

Background

Kubernetes is the application hosting software that Moonlight uses. Kubernetes runs applications on groups of servers. The servers are called nodes. Copies of an application that run on the node are called pods. Kubernetes has a scheduler that dynamically decides which pods should run on which nodes.

Timeline

The original errors on Friday appeared to be connectivity problems with a Redis database. Moonlight's API uses Redis on every authenticated request to validate sessions. We noticed that our Kubernetes monitoring reported that some nodes and pods were unresponsive. Our hosting provider Google Cloud reported networking service disruptions, and we assumed that the outage was the root cause.

As traffic declined over the weekend, the errors seemed to resolve with only occasional problems. On Tuesday morning, the Moonlight website was offline with zero external traffic reaching the cluster. Finding another person on Twitter with similar symptoms led us to believe that Google's hosting was experiencing a networking outage. We contacted Google Cloud Support, who quickly escalated the outage to their support engineering team.

Google's support team identified a pattern in the nodes of our managed Kubernetes cluster. When the nodes experienced periods of sustained 100% CPU usage, the virtual machine would experience a kernel panic and crash.

Cause

The cycle that caused the outage appeared to be:

  1. Kubernetes scheduler assigned multiple pods with high CPU use to the same node
  2. The pods consumed 100% of the CPU resources on the shared node
  3. The node experienced a kernel panic, resulting in a period of downtime where the node appeared unresponsive to the scheduler
  4. The scheduler would reassign all of the crashed pods to a new node, which repeated the same process - and compounded the effects

Initially, only the Redis pod was experiencing this crashing error. But, eventually, all pods serving web traffic were going offline, resulting in a complete outage. Exponential back-off rules resulted in longer periods of downtime.

Fix

We were able to restore website functionality by adding anti-affinity rules to all of our major deployments. This spreads the pods out across nodes automatically, which increases failure tolerance and improves performance.

Kubernetes is designed to be a fault-tolerant hosting system. Moonlight runs three different server nodes for fault tolerance, and we run three copies of every web-facing application. The intention was to have one copy on each node, which would allow two node failures before downtime. However, Kubernetes occasionally scheduled all three website pods on the same node, which created a single point of failure. Other CPU-intense applications (specifically - server-side rendering) were being scheduled onto the same node, rather than separate ones.

A properly-functioning Kubernetes cluster should be able to manage sustained periods of high CPU usage and move pods around to utilize available resources efficiently. We are continuing to work with Google Cloud support to identify and fix the root cause of the server kernel panics.

Conclusion

Anti-affinity rules make web-facing Kubernetes applications more fault-tolerant. If you run a user-facing service on Kubernetes, you should consider adding them.

We are continuing to work with Google to identify and fix the cause of the node kernel panics.

Clients can now buy single job posts

We recently launched membership options for clients. We have added the ability to purchase a single job post without a subscription.

Single job posts do not give access to Moonlight's full functionality, such as search, cold outreach to developers, and direct hiring. But, they are a great way to test Moonlight before committing to a membership.

New homepage

We redesigned the Moonlight homepage! Check it out and let us know if you see any problems.

moonlight-image.png

Updated profile themes

We have updated the default profile and header images on Moonlight! To change yours, go to your profile and click "Edit"!

satellite.png

Important updates to Moonlight's pricing

Today Moonlight is switching from per-hour and per-hire fees to a subscription membership for clients.

For Developers

Moonlight is free. Previously, your hourly rate included Moonlight's 15% hourly rate. Now, that 15% fee has been removed, so your hourly rate has decreased by 15%. You still earn the same amount per hour, but you may wish to tune your hourly rate.

So, if your rate was displayed at $100/hr before, you earned $85/hr after Moonlight's fee. Now, your rate just shows as $85/hour.

For Clients

Before, Moonlight included an hourly fee for all transactions. Hourly rates no longer include that fee, so rates are cheaper across the site.

In addition, Moonlight previously charged $5K per full-time hire. That fee is now waived for subscribers.

To continue using Moonlight to post jobs and hire developers, you'll need to pick a membership plan.

Questions and answers

Do subscribers still have to send payments through Moonlight?

No. But, if you still choose to do so - Moonlight charges a 5% processing fee.

What is happening with open and ongoing projects?

If your project is ongoing, you have been grandfathered into Moonlight's hourly rates. A subscription may save you money - contact us if you want to discuss this.

What is happening with the referral program?

We will be overhauling it in the next few weeks!

Any other questions?

Set up a time to talk with us here!

Updates to our file hosting

This past week, some users have reported issues loading Moonlight's website. The underlying problems seem to be related to HTTP/2 errors while loading static files that we use to show the site in your web browser. Replicating and fixing the issue has proved challenging.

To solve the problem, we are moving all of Moonlight's static assets (such as Javascript files and stylesheets) onto Google's CDN. The assets are being served from a new domain, moonbay.app. Using a different domain stops the browser from sending cookies with the request, which make the requests simpler and faster. Files are now being served directly out of Google Storage, which completely isolates them from our web servers.

If you experience any problems loading our site, please email us at team@moonlightwork.com so that we can help!

No published changelogs yet.

Surely Moonlight will start publishing changelogs very soon.

Check out our other public changelogs: Buffer, Mention, Respond by Buffer, JSFiddle, Olark, Droplr, Piwik Pro, Prott, Ustream, ViralSweep, StartupThreads, Userlike, Unixstickers, Survicate, Envoy, Gmelius, CodeTree