The Modern Marketing Automation Platform

Background

Conversion is building the marketing automation platform of the future. As we have continued to scale our product features, we have also scaled the amount of endpoints reaching over 350 managed endpoints across 10 services. This post will detail the journey we took with service to service communication used to handle >30 million requests a day.
‍
With the introduction of generics, we’ve seen new frameworks like huma and fuego which handle endpoint routing and generating documentation. This blog post details our Service Framework which goes one step further and allows services to call each other without an intermediate representation (and thus build step).

OpenAPI Generated Handlers

Within the software industry debate there is a heavy debate between monolithic architectures and microservices. At Conversion, we believe that services should be “macroservices” which means for basic functions we call a function instead of making a network request but get the benefit of separation for large concepts to keep our codebase manageable. Although we try to reduce “service sprawl”, we’ve seen the business need to separate into a few macroservices due to scaling limitations, domain specification, and centralizing common resources.

Our first approach to service to service communication was to generate handlers based on the OpenAPI specification we had for each service. At this point in our journey, all of our functions were commented in Golang with the documentation that could generate OpenAPI specifications using swag. The natural step was to use a framework to generate client handlers based on this specification.

We accomplished this with go-swagger which allows us to specify templates to generate files with Golang code based on the OpenAPI specification. While this served us well for over a year, our service complexity continued to increase and we saw shortcomings in the lack of strong typing, slow generation times (>2 mins), and poor developer experience requiring developers to regenerate handlers manually. Additionally, engineers needed to context switch between the terminal, repositories, and their IDE — disrupting the coding “flow” and creating a poor developer experience. We had clearly reached the limits of this setup and needed the next evolution.

Generic Schema and Handlers

For a while, there seemed to be no great answer to our problem without adopting heavy handed tools like protobuf / gRPC / bazel. However, we decided that for our scale (~7 engineers) these tools would create a large maintenance overhead and preferred simpler to maintain solutions.

Fortunately, in Go 1.18, Golang introduced generics which are a strong language primitive allowing logic and code to be re-used for different types. This, in combination with the fact that our backend is completely written in Golang and uses Fiber, allowed us to come up with a framework that defines schemas directly in Golang and use them in handlers without reflection or code generation.

At a high level, developers write a schema which defines the types that this endpoint uses. This schema is then used to instantiate a route which requires the handler function to match and accept the specified types. This guarantees via typing that the function is valid to represent this endpoint / schema. Finally, other services have all the metadata required to call this endpoint via the schema and can require a type safe request to be constructed all in Golang.

An example schema would be the following which defines a single endpoint that has a path with the templateId parameters:
‍

package emailschema

type TemplatePathParams struct {
    TemplateId uuid.UUID `json:"templateId" validate:"required"`
}

var (
    TakeTemplateScreenshotV1 = schema.EndpointWithPath[TemplatePathParams, string, EmailApiHandler]{
        EndpointSchema: schema.NewEndpointSchema[EmailApiHandler](
            "TakeTemplateScreenshotV1",
            "/v1/templates/:templateId/screenshot",
            http.MethodPost,
        ),
    }
)

We can then define a type safe endpoint using the following pattern:
‍

func (a *App) TakeTemplateScreenshotV1(ctx context.Context, params emailschema.TemplatePathParams) string {
  return "success"
}

routing.RouteEndpointWithPath(
  emailschema.TakeTemplateScreenshotV1, 
  endpointBuilder, 
  a.TakeTemplateScreenshotV1
)

Finally, we can call this endpoint from another service using a generic handler invocation:

resp, err := emailschema.TakeTemplateScreenshotV1.
  New(app.EmailHandler).
  WithPathParams(emailschema.TemplatePathParams{
    TemplateId: uuid.MustParse('UUID')
  }).
  Do(ctx)

All of this is made possible by our Service Framework! We now have a type safe service endpoint and type safe way to call other services. This is all done natively with generics greatly simplifying our build process (just go build!), improving developer experience with realtime feedback, and using an interface definition language (IDL) that must be correct.
The above example is a slightly abbreviated example of what it looks like to use our service framework. While we cannot open source our entire implementation due to its deep integration with our tech stack, we have created a simple version of this framework in an open source repository at https://github.com/tapp-ai/service-framework-example! This repository includes a simpler version of the machinery we use and a full example of an implementation.

The new Service Framework has been in production for over a month and we continue to migrate services to use the new framework. Beyond the many benefits including no build steps and greatly speeding up the developer experience, we’ve also been able to leverage this new framework to build infrastructure level tools like detailed observability, rate limiting, and request validation.

It’s an understatement to say that this has led to a big improvement in the intangible of “how fun it is to write code at Conversion” while delivering numerous business benefits.

Acknowledgements

The Service Framework is a collaboration and achievement shared by all of the backend engineering team at Conversion. This wouldn’t have been possible without the inspiration and leadership of Tayler who has been the internal champion of the new framework and led many of the early proof of concepts.

This achievement would also not have been possible without the help of Charlie, James, Naasir, and Swamik with their help in migrating our existing and critical endpoints to use the new framework.

If you love tackling interesting technical problems like the ones in this article, Conversion is hiring! We’re constantly working to build new features, improve reliability, and scalability. Please reach out if you’d like to join us on the mission of building the future of marketing.

‍

Background

‍

Conversion is building the marketing automation platform of the future. Powering AI augmented email nurture, a new standard for lead scoring, and integrated product data in their workflows is no small feat and requires solving difficult technical problems. This article dives into how we designed and built the data connector and data synchronization platform — the component responsible for syncing millions of our customers’ most important business records in from and out to their CRM’s every day.

Architectural Considerations

Our urgency for the initial version of the platform and first connector (Hubspot) was astounding — all initial pilots were ready to try our product whenever the first version was ready. At startups, the winners are decided by iteration speed and decision making so the pressure was on.

While it would be easy to lean too far into “build fast and break things”, we prioritized reliability and robustness of the system, while secondarily optimizing for synchronization latency. At Conversion, we understand our customers trust us with some of their most critical business workflows and data, and it could be disastrous if we write the wrong data back to customer CRM’s or miss running an automation for a set of high intent prospects in a campaign.

With this in mind, we designed a system with built in retries, reconciling capabilities, and almost real-time data syncs.

Data Modeling and Orchestration

‍

‍

Data models within different CRMs have complex relationships and dependencies. Our first challenge was transforming and combining high cardinality, heavily interdependent data that varied based on the CRM into our opinionated data philosophy of how you should model marketing customer lifecycle data. 

We opted with running our own in-cluster apache airflow instance. We essentially ran ETLs that would pull in, transform and load data from external connectors into our systems. This was lightweight, had built in system retries, and allowed us to schedule, reconcile, and debug specific nodes in the DAG’s without failing the entire run. This approach also allowed us to design data syncs around the dependencies of external providers — eg custom ordering of nodes based on the external providers entities. 

One unique aspect of our implementation is that our backend systems are all written in Golang! In order to make this work, we use K8s operators running go binaries instead of Python

How We Plug and Play Connectors

There are many sources of data that need to be synced into our systems (we haven’t even talked about data warehouses or our public API). It was top of mind to make it as easy as possible to integrate new connectors into our systems and maintain current ones! 

‍

We leveraged Go’s interfaces to make an object hierarchy that abstracted away writing entire ETL scripts into implementing simple interfaces based on the external API’s. This works by making each DAG node instantiate an instance of the syncer configured with environment variables from that node. That syncer instance handles all generic ETL logic of parallelizing and looping through data. Finally, it calls interface functions that are CRM + external entity specific. 

‍

The end goal with the platform is that if you want to sync another entity for a specific CRM, the core syncing logic is re-usable and the developer only has to implement a interface specific for the data source.

‍

Currently Supported CRM Connectors

For Hubspot we currently sync the following entities: Contacts, Companies, Deals, Calls, Meetings, Notes, Tasks

For Salesforce we currently sync the following entities: Accounts, Contacts, Events, Leads, Notes, Tasks, Opportunities

The tricky part here is that each of these entities could have thousands of custom fields per instance of an entity. Storing such high cardinality data to be easily and efficiently queryable is slated for a future blog post!

Scheduling and Reconciling

It’s inevitable that the syncer will have downtime and our goal is to recover as fast as possible. In order to make this a reality, we have a robust system of scheduling and reconciling.

Diving a bit deeper into the syncer — when a customer installs the Conversion app into their CRM we kick off the inital syncing job. After this, we run a continuous sync every two minutes. Within each of these syncs, we break down the data that needs to be processed into small chunks (<1 MB) which is then stored in a Google cloud bucket.

Chunking the large amounts of incoming data has a two distinct advantages: parallel processing and never losing progress. Whether it is a transient failure, node restart, or bug in a later stage of the syncer, we are able to restart processing for a specific chunk in under 10 seconds helping us drive down the syncing latencies even when there are errors.

With chunking, we now do the following whenever we face an error:

• Retry from the failed DAG node + chunk combination up to 3 times
• Page on-call Conversion engineer if a specific chunk fails after retries
• Reconcile all data from transient errors via a daily cronjob

Overall, this architecture has helped us drastically reduce the time to recovery and ensure customers don’t miss critical updates.

Looking Ahead

We hope this small peak behind the curtain at one of our most critical systems was insightful. We could write pages upon pages of implementation details but in the spirit of keeping the reading brief we’ve spared you guys many of the details that we’ve spent hundreds of hours contemplating!

Getting this far would’ve been impossible without the talented engineers at Conversion who maintain and continuously improve this living, breathing data synchronization system. Especially huge shoutouts to Swamik Lamichane and Naasir Farooqi for being main contributors on the project.

If you love tackling interesting technical problems like the ones in this article, Conversion is hiring! We’re constantly working to build new features, improve reliability, and scalability. Please reach out if you’d like to join us on the mission of building the future of marketing.

Introduction

Just as SEO (search engine optimization) is indispensable for online success, SEO tools serve as your trusty companions in navigating the complex world of digital marketing. These tools simplify the intricacies of SEO, allowing you to save time, reduce guesswork, and gain valuable insights for crafting an effective SEO strategy.

Whether you're striving to enhance your website's visibility on search engine results pages or striving to drive more traffic, SEO tools are your secret weapons.

In this guide, we'll delve into how these tools can revolutionize your online presence and deliver tangible results for your business.

What is SEO?

If you handle any portion of your business online, SEO is one of the most crucial concepts that you must master to have any form of success online. SEO stands for search engine optimization, and to simplify, is the process of ensuring your site ranks near the top of search engine results and bringing traffic to your website.

Search engines like Google, in order to find websites to suggest to users, crawl websites and analyze their content. The engine then uses this analysis to best determine what the site is about, if it’s authoritative, and to which queries it may be relevant. Essentially, the engine attempts to determine if your site is worth suggesting to users.

With SEO, you work to optimize your site to be as worthy of suggestion as possible. This means paying deep attention to your site’s content, loading speeds, keywords, and much more.

Sounds like a hassle? It can be, but regardless, is an incredibly important aspect of any business’s digital marketing strategy.

Why is SEO important for my Business?

Think of it this way, without SEO, your site might as well be invisible in the digital landscape. With 93% of web traffic coming from search engines, unless users have a direct link to your website, it’s almost impossible for them to find you.

Sadly it’s not enough to solely be listed on search engines, as without SEO your site will appear much lower in results. Only around 4.8% of users click on results in the second page of search, meaning you’ll receive few clicks, if any clicks at all, if your site doesn’t rank near the top of results.

Additionally, it’s almost guaranteed that your competition is working to rank on search. If a potential customer searches for a solution that your business provides, and a competitor appears first, they’re very likely to try them and even purchase from them, without even getting the chance to discover your product in the first place.

Without search, you’re missing out on a crucial source of traffic, users, and even potentially income. All this goes to show that SEO is a massively important piece of any marketing strategy.

What Are SEO Tools?

As you may have guessed, SEO is a behemoth of a task, and nearly impossible to figure out all on your own. Yes, you can write great content and hope your site gets clicks naturally, but this may be a long and tedious process without results, meaning less site visitors and a very real possibility of impact to your small business’s bottom line.

Luckily, due to the crucial nature of SEO for any business today, plenty of tools have been developed to help you along your journey. Sure, it’s still going to be a difficult process, but SEO tools can help alleviate a lot of the guesswork and trial and error, as well as provide valuable insight into your SEO strategy. Some are free and some are paid, but they help at every step along the way, with focuses on analysis, research, auditing, and so much more.

Read on for more information on what these tools can do for you, and which big names are currently at the top of our lists.

What can SEO tools do for you?

In the following section, we break down the key aspects of SEO tools and how they can streamline the process for your business. Starting with…

Keyword research

To keep it simple, keywords are terms that are commonly searched on Google (or other search engines). In order for the right customers to come to your website, you want to ensure that keywords that are included in your site’s content are worthwhile. To put it simply, your keywords must be

• Relevant - The keywords must be something that leads a potential customer to visit your site. Think, ‘top camera lenses’ for a photography site, etc.
• Intent based - users use search engines for a variety of reasons, including informational, transactional, and navigational reasons.
• Low competition - remember, plenty of sites are competing for the same customers you are, and thus it’s ideal to find keywords that are feasible to rank for.
• High traffic - Keywords ideally have plenty of searches each month, otherwise there is no point to ranking with them

Luckily, SEO tools have the ability to streamline this process, with dedicated tools to analyze keywords as broadly or as granularly as you would like. Not only can they analyze keywords individually, but they can also find you similar keywords that may fit more optimally within your strategy.

Competitor analysis

As we mentioned earlier, it’s incredibly likely that your competitors are already working on their own SEO, so, it may be (or it definitely is) a good idea to analyze how they gather their traffic. With SEO tools, you can delve deep into your competitors, with decent estimations of their traffic numbers, traffic sources, keywords, backlinks, and more, all available at your fingertips.

Think of it almost as cheating a little bit. You get the opportunity to peel back the curtain of their strategy, and see what works for them and what hasn’t been panning out. Using these tools, it’s trivial to leverage this information to get ahead.

Backlink auditing

One additional vital piece of the SEO puzzle are backlinks. Google, or other search engines, use backlinks to gauge whether your site is a reliable source of information.

Unfortunately, a large number of backlinks isn’t always the best for your site, as each link is given an authority score as well. If a large number of your backlinks come from sites that Google deems “toxic”, your site will be seen as toxic as well, and be lowered in the rankings.

Using SEO tools, you can track each individual site that links to your content, judge their toxicity score, and disavow them if necessary. Of course, this can be done manually, but it’s much easier and more straightforward with a dedicated tool.

Site auditing

Sure, your website can be visually stunning and contain quality information, but what if there are aspects that are hindering your ranking that you are not even aware of? Page speed, duplicate content, and many more invisible factors do directly affect your site’s SEO, and without a dedicated tool, can be hard to handle.

With a dedicated SEO tool, you can get direct insights and suggestions for your site at a glance, making it easier to resolve small issues that may be holding your site back.

Rank tracking

With all these variables and all the effort you’re beginning to put into SEO, how can you keep track of your progress? Most SEO tools offer a dedicated platform to analyze your site’s progress climbing the ranks. From overall site rank stats to keyword ranking over time, to competitive comparisons and more, SEO tools provide a birds-eye view of your SEO’s progress.

Overall, SEO tools provide a myriad of features that streamline the process of optimization. Without tools like these, you’d be reduced to making educated guesses and predictions based on little info.

Next up, we’ll talk about some must-have SEO tools, how they differ, and how they integrate with your business.

Must-Have SEO Tools for Small Businesses

Google Search Console

Price: Free

If you’re going to start anywhere with SEO tools, you might as well start here. Google itself created a well fleshed out, free SEO tool that provides an accurate, straight-from-the-source look at your site’s traffic and rankings.

What Search Console holds above the other tools in this list is that none of the statistics shown here are estimations, as each piece of data is provided from Google itself, the ultimate source of truth.

Google Search Console gives you access to site performance info, including query rankings, page performance, clicks and impression info, click-through rates, and more. On top of that, it also offers site speed analysis, including page-load times as well as other core web vitals, across both desktop and mobile. Not to mention, you request Google site-crawls/indexing, update sitemaps, and remove content from search, all in one place.

Unfortunately, Search Console does not provide much analysis or recommendations for SEO improvement, instead simply provides the data. If you’re looking for keyword suggestions, content insights, or anything else, you might need to look elsewhere.

The takeaway is that Google Search console is a necessity for any website owner, as it provides valuable insights into your SEO strategy with data straight from Google itself. It’s important to use this information in tandem with any other SEO tool to create the fullest picture possible of your SEO performance.

Sure, it doesn’t provide solid strategy, insights, or competitive research opportunities, but it provides an incredibly useful baseline for your strategy, and a great way to track your performance over time.

Pros:

• Accurate data
• Strong performance analysis tools
• Provides crucial functionality for SEO

Cons:

• No competitive information
• Rudimentary data insights
• No content recommendations or research tools built in

Semrush

Price: $129-499/mo

If Google Search Console is too much data and not enough actionable information, Semrush is right up your alley. Semrush specializes in research, competitive analysis and optimization. Think of it as your swiss army knife of SEO strategy and analysis.

When you first open up Semrush, you realize just how dense it is. There are a myriad of tools at your disposal, with everything from in depth keyword research to valuable competitive analysis, to even content writing assistance.

Where Semrush really shines is its competitive research tools. By selecting a few of your competitors, you can analyze everything from rankings to backlinks to traffic to sources. By comparing it to your own site, you can gather insight into their strategy, its strengths and weaknesses, and where you can integrate their findings into your own.

Additionally, Semrush has quality tools built around keyword research, analysis, and planning. With tons of data on each keyword’s intent, traffic, competition and so much more, the often complicated effort of keyword research is simplified dramatically.

This just scratches the surface of Semrush’s tools and what’s available to you, the user. There’s only a couple issues we’d like to note. First of all, nothing is automatic, and the learning curve is definitely still there. It’s almost over-complicated at first glance, yet mastering it may be worth your time if you are truly dedicated to your site’s SEO.

Additionally, it’s expensive, especially for smaller businesses or for personal websites. At over $120 a month at the lowest tier, it’s something that may be too risky or cost prohibitive if you aren’t especially cash flow positive.

Lastly, it isn’t necessarily precise, at least compared to Google search console. When comparing the two, it’s quite common to see discrepancies between the datasets. This is why we recommend you use Semrush as more of a general analysis tool rather than a precise source of truth for your traffic numbers.

Pros:

• Fantastic competitive analysis tools
• Great keyword research and insights
• A myriad of other useful tools for content marketing

Cons:

• Expensive
• Inaccurate compared to search console
• Cluttered

Ahrefs

Price: $99-$999/mo

Initially a backlink analysis tool, Ahrefs has expanded its efforts into on-page SEO, and now is a direct competitor to Semrush. Ahrefs, like Semrush, offers stellar competitive analysis, backlink research and troubleshooting, keyword research tools, and much more. It’s another full fledged SEO toolkit, that while expensive and cluttered, is great at getting you the info you need.

Ahrefs offers competitive analysis, with in-depth data on your competitor’s sites, including their keyword rankings, backlinks, and overall traffic performance. This is super useful when it comes to developing your own strategy, as it offers you a great opportunity to take the best parts of your competitor’s SEO and use it for your own site.

Additionally Ahrefs offers keyword research tools, as well as a myriad of other tools for SEO, like backlink analysis. Having originally been a link building tool, it truly shines here. For example, Ahrefs can help you identify broken links, analyze page depth, organize your site’s linking structure, and much more. Its strength absolutely lies in its lineage.

When it comes to content marketing, Ahrefs includes a unique tool called ‘content explorer’. This tool allows you to enter a topic you’d like to write about, and finds matching content all over the internet. This allows you to discover similar content that has performed well, and gives you an opportunity to emulate it.

Both Semrush and Ahrefs are generally comparable feature-wise and pricing-wise, yet Ahrefs offers a different monetization scheme, based on usage. It’s a credit based system that may be less attractive than a single set price.

Overall, Ahrefs offers another option for SEO research and ideation, that is very on par with Semrush. Accuracy is debatable, yet offers a good directional sense of data that can be helpful for competitive analysis.

Pros:

• Competitive analysis
• Great keyword research tools
• Unmatched link building ability

Cons:

• Expensive, complex pricing scheme
• Accuracy is debatable
• Complex, like Semrush

AI SEO Tools

Unless you’ve been living under a rock, you’re probably familiar with the recent resurgence of AI into the collective consciousness. Since OpenAI released ChatGPT into the wild, nearly everyone has been racing to develop tools that take advantage of the recent advancements in machine learning. Of course, SEO has been included in this golden era of AI.

Think of it this way, what if you could completely automate your SEO strategy? The SEO process is difficult and time consuming, especially for smaller businesses who may not have the time or resources to dedicate to focusing directly on their SEO. Luckily, AI can augment or all-together run your SEO from beginning to end.

Seona

Price: $49.99/mo

We’d love to take a moment to draw some attention to Seona, an all in one SEO tool that acts as your own personal SEO agency. Where other tools like Semrush and Ahrefs do provide great information, it’s still up to you to act upon it. Seona completely automates the entire SEO process automatically.

To start, Seona can automatically audit your website, gather your search rankings and gather insights on your performance in SERPs, or search engine result pages. Using AI, Seona will automatically fix code errors, optimize site performance, write and post blogs, and provide updates all automatically.

Although the tool is still in development and is constantly evolving week over week, it’s a great glimpse into the future of AI search engine optimization. If your business would prefer to hand off SEO and focus on your customers, we think you’d love Seona.

Best Practices for Small Businesses

Now that you’re familiar with some of the best SEO tools available to you, let’s talk about some best practices for putting these to use and for kickstarting your SEO journey.

Start by checking your current performance.

How can you know what you need to improve without looking at how you’re currently doing? Before doing anything SEO related, it helps to get a general understanding of where your site currently stands. Check what keywords you currently rank for, and which ones could use some additional content to bolster in rankings. Check engagement information like bounce-rate and stickiness, and use it to get a good gauge on how your site is performing.

Become the customer

When working to find keywords, you need to put yourself in the shoes of your potential users and customers. Try to think like they do, and ideate on what they may be searching for on the internet.

For example, if you sell camera lenses, your ideal customer is likely a photographer. In this case, you may believe that your customer who may be intending to purchase a lens in the near future, might search for ‘best lens for portrait photography’, or ‘how to take better photos’. They may even search for things like ‘how to pose for portraiture’ or something a little more out of the box. Be creative!

Using this info, you can bring your SEO tool a couple ideas on what verticals your keywords might fall under.

Analyze (and steal from) your competitors

Chances are, you have one or more competitors that have figured out a little more SEO than you have. It’s not a bad idea to use some of their ideas to get off the ground. Use SEO tools to find their traffic sources, find what keywords they’re ranking for and what sites are linking to them. You might be surprised what you can replicate for your own site, and what learnings you can gather from their mistakes.

Create great content that resonates with your audience

In the end, no matter what wizardry you attempt to pull off or however you try to game the system, there’s no substitute for hard work and good content. Try your absolute best to provide genuine value and great content for your audience, and the traffic will follow. It’s a total waste to invest heavily in your SEO if your content isn’t good enough to warrant visitors in the first place.

Conclusion

Regardless of where you are in your business’s journey, it’s never too late to get started on improving your SEO. Sure, the process can be complicated, but the SEO tools we’ve mentioned can drastically streamline the process and help your business begin to climb the rankings. If you’re interested in reading more, check out our blog page. Otherwise, if you’re looking for a tool to completely automate SEO for your business, check out our AI SEO tool, Seona.

‍