Skip to main content

Integrating AI Code Reviews in Azure DevOps & GitHub

· 4 min read
Craig Dempsey
Craig Dempsey
Cloud Devops Engineer @ Digital Reflections

CodeRabbit is an AI-powered code review tool designed to streamline and enhance the code review process. It integrates seamlessly with platforms like GitHub, GitLab, and Azure DevOps, providing context-aware, line-by-line feedback on pull requests within minutes. By leveraging advanced AI models, CodeRabbit not only identifies potential issues such as bugs, security vulnerabilities, and performance bottlenecks but also offers actionable suggestions to improve code quality and maintainability. Its adaptive learning capabilities allow it to evolve with your team's coding standards and preferences, ensuring consistent and efficient reviews over time.

Beyond traditional code analysis, CodeRabbit offers features like auto-generated pull request summaries, sequence diagrams, and real-time chat interactions within the pull request interface. Developers can engage in natural language conversations with the AI to clarify suggestions or request further assistance. Additionally, CodeRabbit supports integration with tools like Jira and Linear, facilitating seamless issue tracking and project management. With options for both SaaS and self-hosted deployments, and a commitment to data privacy through ephemeral review environments and SOC2 Type II certification, CodeRabbit provides a comprehensive solution for modern development teams aiming to accelerate their workflows and elevate code quality.

Setting up CodeRabbit in Azure DevOps​

Note - I'm setting this up in my cloud test Entra ID tenant and Azure Devops Organization.

  • Login to the CodeRabbitAi website.

Logging in to Azure Devops

  • After logging in, you’ll be redirected to a Microsoft Apps consent request screen. If you're not the admin, you'll need to forward this to your administrator for approval.

Admin Consent

  • Once consent is granted, you’ll land on the Onboarding page. Select the Azure DevOps organization you want to integrate with.

  • Head to the Repositories page and toggle the switch to enable CodeRabbit on your desired repositories.

Repositories

Creating a dedicated user for CodeRabbit​

For clarity and auditability, I recommend creating a dedicated user account for CodeRabbit in your environment.

Steps: -

  • Go to Entra ID and create a new user.

Create a new user

  • Confirm the user has been successfully created.

New user created

  • In Azure DevOps, navigate to your Organization Settings > Users, and click Add users to invite the newly created account.

Add a new user

  • Log in as the new user and update the profile picture to use the CodeRabbit logo—this helps make it easily recognizable in PRs.
  • Generate a Personal Access Token (PAT) with the following scopes:
    • Work Items: Read, Write, Manage
    • Code: Read, Write, Manage
    • Build: Read

Create a new Personal Access Token

  • In the CodeRabbit web UI, go to Organization Settings > Azure User, and paste the PAT token.

Azure DevOps User

Setting up CodeRabbit in GitHub​

Login with GitHub

  • Authorize CodeRabbit to access your GitHub account.

Authorize CodeRabbit

  • Select the GitHub organization where you want to install it.

GitHub Organization

  • Choose whether to install CodeRabbit for all repositories or only specific ones.

Install & Authorize

Taking it for a test drive​

Time to see it in action! I used the creation of this article as a test case.

  • I committed changes to a feature branch and pushed it to GitHub.
  • Then, I created a Pull Request.
  • Here’s what happened next:

Create a Pull Request

  • After a short wait, CodeRabbit kicked in with a few nice surprises:

  • It generated a PR Summary.

CodeRabbit Summary

  • It also created a walkthrough, a summary of changes, and even a sequence diagram—very slick!

CodeRabbit Walkthrough

  • Most impressively, it reviewed my code. I received both nitpick-level suggestions and an actionable comment.

CodeRabbit Comment

I'm only scratching the surface of what CodeRabbit can do, but I can already see the potential. I'll continue experimenting and plan to follow up with a deeper dive after using it for a bit longer.